The connection between stream restoration and sediment budgeting runs both ways: stream restoration is proposed as a means to reduce sediment yields, but an accurate understanding of sediment supply is necessary to design an effective project. Recent advances in monitoring technology, geochemical techniques, high-resolution topography data, and numerical modeling provide new opportunities to estimate sediment erosion, transport, and deposition rates; upscale them in a geomorphically relevant fashion; and synthesize sediment dynamics at watershed scales. For practical application at large scale, watershed models used to predict yield often do not resolve lower-order channels, leaving an essential “blind spot” regarding sediment processes. We illustrate the challenges and emerging approaches for estimating sediment budgets using examples from two very different physiographic settings: the Mid-Atlantic Piedmont and the agricultural plains of southern Minnesota. We highlight common challenges and themes in defining an effective watershed sediment model. In both cases, reliable estimates of sediment yield depend essentially on the accurate identification of sediment sources and sinks and, hence, require careful delineation of landscape units and identification of dominant sediment sources and sinks. The primary elements needed to bridge the gap between sediment budgeting, watershed modeling, and stream restoration are (1) specificity regarding location, mechanism, and rates of erosion, (2) accurate accounting of sediment storage, (3) appropriate methods for upscaling local observations, (4) efficient means for incorporating multiple lines of evidence to constrain budget estimates, and (5) stream restoration methods that incorporate sediment supply in assessment and design procedures.