Stabilized aggregate containing Special cement with desulfurization residue is characterized by long setting time, slight expansion, and retarded inducing strength, therefore, research was conducted on its performance. Research results show that when on 28d and compared with stabilized aggregate of PC 32.5 cement, the stabilized aggregate of special cement’s unconfined compressive strength increased by 17%, dry shrinkage energy anti-cracking coefficient by 78%, whereas resilient modulus and splitting strength indicated small changes, whilst the performance of special cement materials showed vast improvements through significant increases in the stabilized aggregate’s mechanic and shrinkage performances. The strength of stabilized aggregate from special cement containing desulfurization residue mainly comes from AFt, Ca(OH)2 and the C-S-H gel formed in the hydration of the special cement and also with the continuing dissolution of CaSO3. The resultant new phase of C3A•CaSO3•11H2O is formed after the 7d hydration period. Desulfurization residue in cement consists of a significant amount of Ca(OH)2, therefore, the alkali content in the hydration phase is maintained at a very high level, which affects the consistency of ettringite through the formation of higher-quality needle-like crystals during hydration, which in turn is beneficial to reduce the base material’s self-shrinkage capability and improve its anti-cracking performance. At the same time, stimulated by Ca(OH)2-CaSO4, the hydration of fly ash and ground granulated blastfurnace slag is also improved, which is advantageous for further improving the cement’s strength and anti-cracking performance at a later stage in the curing process.