Increasing incidences of eutrophication and groundwater quality impairment from agricultural nitrogen pollution are threatening humans and ecosystem health. Minimal improvements in water quality have been achieved despite billions of dollars invested in conservation measures worldwide. Such apparent failures can be attributed in part to legacy nitrogen that has accumulated over decades of agricultural intensification and that can lead to time lags in water quality improvement. Here, we identify the key knowledge gaps related to landscape nitrogen legacies and propose approaches to manage and improve water quality, given the presence of these legacies.