What case are you using. Air cooled or aio. Chassis fans etc. What is your CPU temp? Have you altered the fan curve. It's probably cooler under stress because your fans are kicking in. If you alter the fan curve you probably be able to get more even cooling.