In desert regions, railway operations have been confronted with the adverse impacts of wind-blown sand. Timely assessment of the fouling degree of sandy ballast is essential for track maintenance and operational safety. To provide a rapid, non-contact, and low-cost measurement approach, this study proposes an image-based framework for sandy ballast fouling assessment. A dataset of 6260 sandy ballast images has been established under controlled laboratory conditions and linked to four reference fouling grades defined by sand-content ratios. On this basis, the traditional ResNet50 is enhanced with a Morphology Attention (MA) module and a Grouped Squeeze-and-Excitation (GSE) module to develop the MAGSE-ResNet50 model, improving boundary-sensitive and region-aware feature extraction for complex ballast images. Comparative experiments, ablation studies, repeated runs with different random seeds, and robustness tests under field-like perturbations were conducted. The experimental results demonstrate that the MAGSE-ResNet50 model improves classification accuracy with only marginal increases in parameters and computational complexity. The validation dataset achieved an average accuracy of 99.15% with a standard deviation of 0.09, outperforming the original ResNet50 by 3.04 percentage points and demonstrating clear advantages over other comparative algorithms. Under five field-like perturbations, accuracies remained between 96.48% and 98.51%, and the average test accuracy reached 98.44%. Visualization results further show that the model equipped with MA and GSE attention mechanisms can more effectively focus on fouling-related regions and ballast-sand boundaries. These results demonstrate the feasibility of the proposed method and its potential as a rapid tool for sandy ballast fouling assessment in wind-blown sandy railways.