Ocean warming is commonly considered unable to excite significant long-term trends in polar motion. Here, however, we argue that this assumption needs to be revised. We demonstrate that steric sea level rise leads to a distinct pattern of horizontal mass redistribution within ocean basins and hence to ocean bottom pressure changes that alter Earth's inertia tensor on decadal and longer time scales. Based on Earth system model simulations, we estimate that ocean warming leads to polar motion of 0.15 to 0.20 milli-arcseconds per one millimeter of thermal sea level rise. This is equivalent to a polar motion rate of about 0.47 milli-arcseconds per year towards 155°W to 160°W for current projections of steric sea level rise during the 21st century. The proposed polar motion signal is therefore not negligible in comparison to other decadal and secular signals, and should be accounted for in the interpretation of polar motion observations.