Ongoing declines of large-bodied frugivores limit the dispersal of large-seeded plants, contributing to their (local) demise and ‘downsizing’ of seeds across assemblages. However, the extent to which human pressure leads to contemporary seed downsizing, and whether extinct megafrugivores have left imprints on seed size, remains unclear. Here, we integrate trait and distribution data for 2852 endozoochorous plant species, 48 extant and 15 extinct frugivore species across 361 assemblages on Madagascar. Using structural equation models, we show that assemblages with higher human footprint, a cumulative index of human pressure, have smaller maximum seed sizes, especially through downsizing of extant frugivores. Furthermore, among assemblages with ‘mega-seeded’ plants (i.e., seeds that cannot be swallowed by any extant Malagasy frugivore), larger seed sizes are associated with larger past megafrugivores, reflecting the legacy of past interactions. Human-driven seed downsizing highlights broader implications in erosions of important ecosystem functions such as forest carbon storage.