Arid regions in Northwest China are confronted with the dual challenges of river sand (RS) scarcity and accumulation of industrial solid wastes, underscoring the urgent need to develop alternative aggregates such as dune sand (DS) and ferrochrome slag (FS). This study completely replaces RS with DS and FS in a calcium sulfoaluminate (CSA) cement system to investigate the influence mechanisms of complex hydration products on drying shrinkage. Based on QXRD experiments and GEMS thermodynamic modeling, the hydration process and evolution of elastic parameters in the CSA-DS-FS ternary system are accurately characterized. A physics-based model, incorporating both elastic and creep strains, is developed to predict drying shrinkage. Results show that composite aggregate replacement significantly intensifies drying shrinkage while reducing matrix stiffness. The 30DS70FS mix exhibited a 79.5% increase in shrinkage compared to the RS reference, as the low stiffness and high capillary water content of the aggregate offset its enhanced hydration degree. In the 50DS50FS mix, the promotion of hydration was significantly inhibited, leading to a 21.7% decrease in elastic modulus and a 203.4% increase in 28-day shrinkage. The proposed model accurately predicts long-term shrinkage (error < 5.0%) by incorporating material stiffness, hydration products, and internal relative humidity. Findings reveal that creep strain increasingly dominates long-term deformation, rising from a 28-day maximum of 16.7-36.1% by 350 days. This research elucidates the shrinkage mechanisms of green solid waste aggregates in CSA systems, providing a valuable theoretical basis and methodological support for the development of sustainable building materials.