The semi-arid regions of the world are populated by highly specialized groups of organisms that can cope with the harsh climatic conditions. However, the climate and, as a result, the floristic composition have changed in these regions in recent decades. Dryland regions throughout the world host biological soil crusts, which colonize the uppermost soil layer. This superficial growth facilitates the utilization of imaging methods for monitoring purposes. In this study a deep-learning model called Joint Energy-Based Semantic Segmentation that enables robust analysis of images captured over long periods of time is proposed. It is shown how biological soil crusts have changed over time in two very different areas (Succulent Karoo, South Africa and Colorado Plateau, USA) with an accuracy of 91 % and 77 %, respectively. This provides a detailed analysis of the complex interactions between the individual taxa and external climatic influences. The results show that conditions of extreme drought led to degradation and that the soil crust organisms were unable to fully recover from this during wetter periods. On both sites, biocrusts degraded within the last 1-2 decades. A detailed time series analysis of the interactions between the occurring taxa, using time lagged cross correlation and transfer entropy metrics, identified Psora sp. and Fulgensia sp. as key indicator species, as they were highly reactive to climate alterations, and therefore inform about biocrust degradation already at an early state. This study demonstrates how modern image-based deep learning methods enable a very detailed analysis of the development of the world's dryland flora.