Generally, how ecological restoration projects affect soil in desert steppe open-pit mining areas is not comprehensively evaluated. This study focused on an open-pit coal mine on the Loess Plateau, Inner Mongolia, China, examining soil physicochemical properties and nematode community composition under three treatments: phytoremediation, natural restoration, and a native vegetation treatment. High-throughput sequencing yielded a total of 141,318 soil nematode DNA sequences belonging to 16 genera. Our analysis suggests that the genera Aphelenchoides, Paratylenchus, Tripylina, Oxydirus, Cervidellus, Nothacrobeles, and Acrobeles may be key indicators of nematode communities in ecosystems that have been damaged by open-pit coal mines, as suggested by LEfSe multi-level discriminant analysis. Redundancy analysis indicated that pH, electrical conductivity, organic matter content, total carbon, total nitrogen, nitrate, and ammonium nitrogen may have been key limiting factors in the soil restoration process. Under the phytoremediation treatment, soil physicochemical properties, diversity, community composition, the relative abundances of different trophic groups, and ecological indices were significantly improved relative to natural restoration. Phytoremediation is very likely to offer greater potential for future recovery than natural restoration. This study helps to deepen the understanding of the restoration mechanism of desert steppe mining areas.