Land surface temperature (LST) is an important parameter in the surface system with drastic variation in spatial and temporal domains. The protection of the ecological environment in mountainous areas and the monitoring of natural disasters require the support of surface temperature data with high spatiotemporal resolution. LST downscaling is an effective method to improve the spatial and temporal resolution of remote sensing LST data. However, at present, the LST downscaling research mainly focuses on plain and urban areas, while the area of rugged terrain is less studied, and the accuracy of LST in rugged terrain is lower than in plain and urban areas. In the few studies that discuss auxiliary parameters for LST downscaling in rugged terrain, only elevation is considered as an auxiliary parameter. In this study, we selected parameters that have evident correlation with LST as potential auxiliary factors and discussed the benefits of adding shortwave radiation to the LST downscaling process. We chose four scene images in the Beijing suburbs and the Loess Plateau and conducted the LST downscaling experiments. In this study, we used the Taylor expansion method for LST downscaling. We selected Landsat 8 and MODSI LST data as fine and coarse study datasets, respectively. The results show that the accuracy of LST downscaling in rugged terrain areas can be improved by using elevation and shortwave radiation as auxiliary factors, and the benefits of shortwave radiation is independent of that of elevation. Therefore, it is suggested that these two parameters be simultaneously used to achieve the best LST downscaling result over rugged terrain areas.