40C when idle doesn't seem that unreasonable, although I'm not 100% certain about typical temperature on the 3080. 55C under gaming load also seems like a reasonable number.
My guess is that this is probably a Windows or Nvidia measurement/reporting bug. If you're on Windows 11 22H2, the following article may be relevant:
Nvidia: 100% GPU usage in Windows 11 Task Manager is a known issue
Another possibility is something like crypto mining malware (although relatively low temperatures would suggest that it's not doing anything particularly heavy). Run a virus scan with either Microsoft Defender or your AV of choice, and also a Malwarebytes scan.
Thanks and very helpful. The article was interesting but perhaps wasn't the problem (new driver and I was having low-moderate power usage). I ran AV and Malware and nothing found (I wondered the same thing about someone using my card, but I agree, low temp tells us the load is not high).
I will keep an eye on it and see if an update fixes it. I welcome any other ideas as I don't have any more.