Change detection (CD) of buildings in high-resolution remote sensing images is crucial for urban planning, disaster assessment, and security surveillance, with key challenges being variations in lighting conditions and the diverse shapes and sizes of buildings. Although numerous convolutional neural network (CNN)-based CD methods have been developed, they often ignore the spatial and contextual information between pixels. In this paper, we propose a spatially and contextually aware Siamese network (SCASN) that utilizes the interaction between graph neural networks (GNN) and CNNs, which exploits the spatial and contextual information through a GNN-and-CNN based parallel hybrid backbone (PHB) network. The proposed PHB adaptively extracts precise positional and spatial information from the input image by fusing local detail and global positional features. Furthermore, we designed a multi-scale integration module that uses dynamic pyramid split attention to address the absence of contextual information at multiple scales by effectively integrating spatial information across different levels. Finally, the high-level features are further refined and processed through a transformer model, which generates the final output maps. Additionally, a unique dataset for building change detection (PD-CD) in plateau and desert environments has been created, capturing specific climate and environmental characteristics distinct from typical urban and suburban datasets. PD-CD comprises 4,095 pairs of bi-temporal images at a spatial resolution of 0.5 meters, each with a size of 256 x 256 pixels. Experimental results on PD-CD and three other datasets demonstrate that the proposed network substantially outperforms several other attention- and transformer-based methods.