We present a new strategy to validate modeled spectral irradiance of shallow cumulus cloud fields in a polluted background with airborne measurements. The concept is based on a spectral distinction of effects associated with heterogeneous clouds, aerosol particles, and surface albedo. We use measurements from the Gulf of Mexico Atmospheric Composition and Climate Study, conducted in the urban‐industrial Houston area. Modeled irradiance fields were obtained from extensive three‐dimensional radiative transfer calculations applied to the output of large eddy simulations. We show that the measurements below clouds or cloud gaps can only be reproduced by the calculations when including the aerosol radiative effects. The technique enables the derivation of measurement‐based spectral forcing and absorption of the cloud‐aerosol system which will help substantiate model calculations. At 400 nm wavelength, the inclusion of aerosol increases forcing of the cloud‐aerosol system by 8%, and absorption by 20%.