Background A phytogenic mound is a special microtopography that forms under a perennial plant canopy in erosion-affected areas. These mounds result in spatial microenvironmental heterogeneity and thus are important factors in determining plant community composition. Aims We assessed whether and how plant composition and diversity differed on different parts of mounds in water erosion-affected areas and evaluated which environmental variables were related to the compositional difference. Methods We compared plant community composition and diversity on the upslope (UP) and downslope (DN) parts of mounds and inter-canopy surfaces (IS) along four slope gradients. Results On slopes < 46.6%, vegetation cover, biomass, density and diversity of plant communities did not significantly differ between UP and DN. However, these plant community attributes were higher UP than in IS and DN on steeper slopes (> 46.6%). On such slopes phanerophytes and chamaephytes occurred only UP. Sediment accumulation, light intensity, soil moisture, and surface hardness were significantly correlated with the community composition on mounds, while soil N, P and K showed a weak relationship. Conclusion Only the UP of mounds on steep slopes maintained high plant diversity in water erosion-affected areas, which is especially important for shaping plant community patterns on seriously eroded slopes where plant colonisation is limited.