To define the role of organic or inorganic fertilizers in the stability and sustainability of soil organic carbon (SOC) and crop yield, we examined changes in SOC and crop yield, as well as their correlation, in nine different fertilization treatments (sheep manure [M], sheep manure and urea [MN], sheep manure and diammonium phosphate [MP], sheep manure, urea, and diammonium phosphate [MNP], urea and diammonium phosphate [NP], urea and potassium sulfate [NK], diammonium phosphate and potassium sulfate [PK], diammonium phosphate, urea, and potassium sulfate [NPK], no fertilizer [CK]) from 2015 to 2018 on the Loess Plateau. The organic carbon content was higher in organic fertilizer (M, MNP, MP, MN) than in inorganic fertilizer treatments (NP, NPK, NK, PK) and the control. Crop yield was higher for organic than for inorganic fertilizers, whereas that of the control was always lower than those in the fertilization treatments. The highest crop yield was obtained in MNP. Crop yield and soil organic matter quality score correlated positively, with correlation coefficients ranging from 0.5994 to 0.7151. Our results provide a theoretical basis for organic and inorganic fertilizer application in dry farming areas on the Loess Plateau and aid the long-term healthy development of soil in cultivated areas.