Agricultural drought (AD) refers to a decline in crop yield as a result of a decrease in soil moisture, which is closely associated with land use type and soil texture. In semi-arid climates, however, it is unclear what role land use type and soil texture play in AD. Here, soil water deficit index (SWDI) considering the physiological state of vegetation is used to characterize AD. This study systematically investigated the effects of land use type and soil texture on the evolution of AD, and also analyzed the temporal changes of soil moisture, water deficiency in the vadose zone, and precipitation at annual and monthly scales. The results indicate that there is a broad similarity between the annual variation trends of soil moisture and precipitation in the Ziya River Basin (ZRB), while water shortages in the vadose zone are opposite. It is evident that SWDI has good applicability in characterizing AD in ZRB by comparing it with meteorological drought (MD). For impacts of different land use types and soil textures, Forest and Grassland exhibit greater drought resistance with the lowest duration (D, 4.12 and 3.92) and severity (S, 7.76 and 7.93), whereas Cropland, Waters, and Urban Land (D > 5.39; S > 11) are more susceptible to severe droughts. For soil textures, Sand (D, 4.56; S, 8.32) and Loam (D, 5.09; S, 10.32) are more prone to AD than Clay (D, 3.24; S, 5.13). Overall, these findings contribute to understanding the role of the land use types and soil textures in the evolution of AD in semi-arid regions and provide practical recommendations for early identifying AD.