Automatic identification of sinkhole hazards is crucial for ensuring the safety and stability of human lives, infrastructure, and ecosystems. Deep learning has demonstrated significant potential in autonomous aerial vehicles (AAVs) low-altitude remote sensing monitoring of sinkholes due to its high scene fine-grained comprehension. However, constrained by the micro-relief characteristics and spatial heterogeneity of loess sinkholes, existing models suffer from the edge confusion and missegmentation of homogeneous internal regions. This fundamentally stems from shallow convolutional kernels failing to effectively decouple nonlinear coupling mechanisms in high-dimensional feature spaces. In this context, we propose a multilevel frequency-domain transformation network named CA-KMTNet. The network employs compute unified device architecture (CUDA)-accelerated Kolmogorov-Arnold networks (KANs) to enhance nonlinear learning capabilities for loess sinkholes. To prevent overfitting, wavelet transforms are integrated into each encoder level, while KANs are embedded in the shallow layers to provide high-dimensional multifrequency spatial features. Subsequently, inverted residual convolutional structures are introduced during deep encoding to optimize edge pixel identification for small loess sinkholes. Experimental results demonstrate the promising performance of the proposed method. On the Baijia gully dataset in the Loess Plateau, the proposed model achieves 15.90%, 19.19%, and 15.90% improvements over UNet in F1-score, intersection over union (IoU), and Kappa metrics, respectively. To further verify the generalization ability of CA-KMTNet, this study tests the previous Baijia gully dataset and the Laozi gully dataset. CA-KMTNet shows stronger spatiotemporal transferability and performance advantages. These findings highlight the identification potential of the proposed model for loess sinkhole hazards and provide new insights for integrating the KAN structure into semantic segmentation tasks.