We investigate the relationship between solar irradiance and cosmogenic isotope variations by simulating with a flux transport model the effect of solar activity on the Sun's total and open magnetic flux. As the total amount of magnetic flux deposited in successive cycles increases, the polar fields build up, producing a secular increase in the open flux that controls the interplanetary magnetic field which modulates the cosmic ray flux that produces cosmogenic isotopes. Non-axisymmetric fields at lower latitudes decay on time scales of less than a year; as a result the total magnetic flux at the solar surface, which controls the Sun's irradiance, lacks an upward trend during cycle minima. This suggests that secular increases in cosmogenic and geomagnetic proxies of solar activity may not necessarily imply equivalent secular trends in solar irradiance. Questions therefore arise about the interpretation of Sun-climate relationships, which typically assume that the proxies imply radiative forcing.