Understanding nonlinear interactions and identifying threshold effects among ecosystem services (ESs) are crucial for addressing sustainability challenges in ecologically vulnerable subtropical watersheds. Previous studies have primarily focused on upper constraints, often neglecting ecological risks associated with minimum thresholds. We developed a threshold-based ES optimization framework to quantify the spatiotemporal nonlinear interactions among four key ESs-water yield, nitrogen retention, soil retention, and carbon sequestration-in China's subtropical coastal Jiulong River Watershed. We proposed an innovative dual-constraint approach to jointly optimize ES synergies while identifying risks associated with breaching ecological redlines. Our results revealed that ES interactions exhibited nonlinear trajectories (e.g., unimodal, saturating, or monotonic), with critical thresholds marking transitions between synergies and trade-offs. These interactions either shifted toward trade-offs or reached plateaus. Anthropogenic activities delayed these thresholds and reduced ES provision. The dual-constraint framework exposed inherent trade-offs: factors achieving maximum ES values under upper constraints simultaneously exhibited lower values falling below ecological baselines, revealing widespread latent vulnerabilities across the watershed. Socio-economic factors universally suppressed ES capacity, especially in downstream urbanized plains. Interactive effects were found to exist among socio-environmental factors, particularly between meteorological and economic factors. Critically, maintaining these drivers at intermediate levels may maximize synergy potential and facilitate win-win outcomes. By integrating nonlinear dynamics with local environmental policy, this study presents a scalable framework to harmonize ecological security and sustainable development in rapidly urbanizing coastal watersheds.