A compilation of the apatite and zircon fission track ages of the Alpine chain points to markedly different patterns of cooling and exhumation of the Eastern Alps compared to the central and Western Alps. The site of exhumation and shortening in the Western Alps migrated outward, whereas it was more stationary in the Eastern Alps, where it created a narrower metamorphic belt. A correlation of these observations to the deep structure of the orogen suggests that north directed, lower crustal wedging induces northward propagation of the deformation and exhumation front in the middle and upper crust of the central and Western Alps. The absence of such lower crustal wedges in the Eastern Alps does not induce a similar shift of deformation but rather a long-term localization of shortening and exhumation in one and the same area, namely, the axial zone of the orogen. A critical review of the temporal correlations between inferred changes in the erosional efficiency of the Alps and tectonic phases points to a negative correlation, and hence no erosional control on the shifts of the deformation front of the Alps. Therefore, we conclude that changes in the deep structure and rheology of the orogen, instead of changes in its erosional efficiency, exerted the prime control on the lateral growth of the Alps.