The lateral variations in the fluid outer core are believed to be very small from fluid dynamics calculations. Seismological studies on the issue have been limited and controversial. A great challenge is to sort out influences from heterogeneity in the mantle or the inner core. Using high-quality earthquake waveform doublets, we found that waves passing through the fluid core over a few years are significantly more variable than those passing through the mantle only. We interpret the temporal variability as the result of the fluid motion of the heterogeneous materials in the outer core. The level of heterogeneity in the fluid outer core is constrained to be ±0.022 s (95% confidence) in seismic travel times through the core, negligible for most seismological studies. However, the estimated velocity perturbation, about 10⁻³ for small-scale heterogeneity (10 km) or 10⁻⁴ for large-scale heterogeneity (1000 km), borders or exceeds the high-end estimates of the lateral variations that can be supported by dynamic forces within the fluid core. The source of the heterogeneity is not clear at present.