So in BIOS 0801 and 0901, with AMD NVME raid enabled (2 drives) it messes up other PCIe SSD (Intel PCIe card) control. Inside BIOS this shows that the Intel drive vanishes from the selectable "other boot drives" when one enables the AMD raid (as does the NVMe drives), while the Intel bios diagnostic still works inside the zenith bios. Intel drive itself is still not listed inside the raid setup interface. Windows then won't be using Intel drivers and the Intel diagnostic drive software won't work, device manager shows the drive as AMD-RAID INTEL... most likely this also happens if one has 3rd m.2 drive that is not in the raid or u.2 drive.
There is the PCie slot setting for nvme raid but it is at a default value (x8). I haven't tested the SSD in a chipset controlled PCIe slot, then again those are only gen 2. I have this feeling that the PCIe SSD might not be getting the best performance under the amd raid layer... but that needs to be tested, with windows updates also slowing disk IO.
So new Windows version available in 48 hours or so, so I decided to dig a little deeper, before clean install, also Contacted AMD support, to find out what they have to say if anything.
The web UI raidxpert2 shows the PCIe Nvme (not m.2) in array 2 available options are JOBD, raid0 and raid1, while the BIOS version doesn't list the drive but the array 2 was visible (offline).
One can delete the array, the drive still shows in the raidxpert2 as 2.1 (0.1 and 1.0 for the M.2 raid), and windows don't detect.
Now windows won't detect any of the drives because at least the AMD bottom raid drivers screws things over, took a while to figure out why I can't get the drivers to show up when Nvme Raid is disabled in BIOS, tried installing/uninstalling amd driver packages (clearly not uninstalling everything). Had to manually set to generic drivers for them to show up. First AMD NVMe raid drivers were installed during Windows install.
So since raid is disabled the partition table "vanished" on the drive, but once the AMD Nvme raid drivers were changed testdisk did manage to find the partition table.
little more testing...
Windows 10 install on the NVME not in the NVMe Raid.
It seems one can use the NVMe drives own controller drivers over the AMD bottom device driver (which are needed for raid 0/1/10) , but the drive still gets placed under the drivexprert2 array as legacy/offline. Trying to install the AMD drive package (latest) blue screens windows … if OS is installed on the NVMe. The driver package install won't run if not all NVMe drives are using the bottom device driver (complains about OS being on NVMe). Even windows installation will crash after installing files with only AMD NVMe drivers loaded, array 2 will be offline+failed in BIOS.
With raidxpert2 installed manually, changing the controller back to Intel driver makes the legacy raid go offline and then the array can be removed. The max latency seemed to drop a little, but still it's under some nvme raid layer, since intel toolkit won't work. Somehow being OS drive and using the amd bottom device driver for the lone intel NVMe causes issues, no such issue before when OS was on old SSD. Now computer got stuck on reboots when using the amd driver.
winsat disk test (Intel 900p)
(not OS drive) NVMe raid disabled with Intel controller driver. Average Read Time with Sequential Writes 0.021 ms Latency: 95th Percentile 0.038 ms Latency: Maximum 0.104 ms Average Read Time with Random Writes 0.021 ms
(OS drive) NVMe raid enabled with Intel controller driver. Average Read Time with Sequential Writes 0.022 ms Latency: 95th Percentile 0.040 ms Latency: Maximum 0.129 ms Average Read Time with Random Writes 0.022 ms
AMD bottom driver (OS) Average Read Time with Sequential Writes 0.052 ms Latency: 95th Percentile 0.064 ms Latency: Maximum 31.889 ms Average Read Time with Random Writes 0.058 ms ...oh and 16.0 random read is 600MB/s slower than the 2 previous ones.
Crystaldiskmark 4KiB Q8T8 and Q1T1 seems to halved with bottom device driver, seems to be driver cap since the NVMe RAID get same results, apart from Q1T1 which is worse on the RAID.