Yes, there is a way to deactivate one of the M.2 drives assuming you are using M.2_1 and M.2_2. I am not sure about slots 3 and 4. Unfortunately I doubt there is an automatic mode.
I am not sure if the same setting is in your BIOS but if my Onboard Devices Config > CPU PCIE Config Mode is NOT set to PCIEx16_1 = PCIEX_2 + M.2_2 the system only sees one of the NVMe drives. I have to set it to PCIEx16_1 = PCIEX_2 + M.2_2 in order for it to see the other NVMe drive.
Seems to me it would probably be easier to configure Windows so it doesn't mess with the drive at all in the Disk Manager.
Press Windows + X and select the Disk Manager option; After loading all connected disks, right click on the NVMe drive you want to disable and go to Properties; Click on the Hardware then find the NVMe drive you want to disable in the list. Double click on the drive to bring up the drive properties and choose Driver and choose Disable Device.
NOTE: You won't be able to do this on the C (Windows 11) drive but you should be able to do it on the other NVMe you don't want Windows messing with. Windows should now ignore the drive.
IMPORTANT!!!! Do a backup before you try this. I recommend AOMEI Backupper which is free for personal use.