Vegetation and surface deposits in semi-arid shallow-covered areas strongly mask bedrock spectral signatures, challenging conventional remote sensing lithological classification. This work proposed a deep learning framework named Multi-scale Temporal Dual-Attention Network (MSTDA-Net), which was designed to extract implicit lithology-vegetation relationships from multi-source time-series data. MSTDA-Net integrates three core modules: multi-scale temporal convolution, temporal attention (which weights the importance of different time phases), and channel attention (which weights the importance of different input features). These modules jointly capture local phenological patterns, emphasize key phenological phases, and enhance discriminative features. Using 12-phase (one phase per month) Sentinel-2 multispectral, Sentinel-1 SAR, and derived vegetation indices, a 25-dimensional monthly time-series dataset was constructed. Experiments show that MSTDA-Net achieves an overall accuracy of 97.14% and a Kappa coefficient of 0.970, outperforming all benchmark models. Ablation studies confirm the necessity and synergy of the three core modules, and feature importance analysis reveals that the model focuses on peak-growing-season signals and vegetation-sensitive features, aligning with geological principles. This work demonstrates that deep learning-based mining of vegetation phenology from multi-source time-series data can effectively infer lithology under shallow cover.