Grazing affects soil organic carbon (SOC) through plant removal, livestock trampling, and manure deposition. However, the impact of grazing on SOC is also influenced by multiple factors such as climate, soil properties, and management approaches. Despite extensive research, the mechanisms by which grazing intensity influences SOC density in grasslands remain incompletely understood. This study examines the effects of varying grazing intensities on SOC density (0-30 cm) dynamics in temperate grasslands of northern China using field surveys and experimental analyses in a typical steppe ecosystem of Inner Mongolia. Results show that moderate grazing (3.8 sheep units/ha/yr) led to substantial consumption of aboveground plant biomass. Relative to the ungrazed control (0 sheep units/ha/yr), aboveground plant biomass was reduced by 40.5%, 36.2%, and 50.6% in the years 2016, 2019, and 2020, respectively. Compensatory growth failed to fully offset biomass loss, and there were significant reductions in vegetation carbon storage and cover (p < 0.05). Reduced vegetation cover increased bare soil exposure and accelerated topsoil drying and erosion. This degradation promoted the downward migration of SOC from surface layers. Quantitative analysis revealed that moderate grazing significantly reduced surface soil (0-10 cm) organic carbon density by 13.4% compared to the ungrazed control while significantly increasing SOC density in the subsurface layer (10-30 cm). Increased precipitation could mitigate the SOC transfer and enhance overall SOC accumulation. However, it might negatively affect certain labile SOC fractions. Elucidating the mechanisms of SOC variation under different grazing intensities and precipitation regimes in semi-arid grasslands could improve our understanding of carbon dynamics in response to environmental stressors. These insights will aid in predicting how grazing systems influence grassland carbon cycling under global climate change.