Is 4x GPU with a Samsung m.2 NVMe drive supported?

[slp_pleh]

Hello,

I have an Asus Rampage V edition 10 board with an Intel Core i7-6850K (40 pcie lanes), four (4) Nvidia 1080 GTX FE GPUs, and a Samsung 950 pro m.2 NVMe flash drive.

When I boot the system up on an SSD disk (NOT on m.2), the pcie slots negotiate to 16/8/8/8 and it works fine.

When I boot the system from the m.2 NVMe drive, the pcie slots negotiate to 8/8/8/8 but the system will not detect the m.2 flash drive.

The user manual, on page ix it states:

40-LANE CPU:
- In 4-Way con guration, if the PCIEX16/X8_1 is used in x16 mode, both U.2 and M.2 ports will be disabled.

So it seems I've hit a bug, or I need to adjust a setting in the BIOS.

Does anyone have any suggestions? The Asus tech support person I spoke with said he would open up a ticket (after I asked him to) with their engineering team but I haven't heard back in almost a week. He seemed to think this wasn't supported but I got the feeling he just wanted to get off the call with me.

Thanks for any help!

[CrusherXL3D]

Thank you.It really works perfectly know. The M2 port speed seems to be slower than on the U2 port.I recognized this by running the magicuan benchmark twice. First at the M2. Result: around 1200 MB/s read. On the U2 port: Full 2500 MB/s! Together with the CMR boot turned off, the system boots now in no-time. Maybe the U2 port handles the 4 PCI lanes topic better as it was made for drives tht can use x4 pci. I dont know. The only thing I know that I finally have my system up and running with 2xTitan X and 2x1080s. Its the perfect rendermachine now for me.

I would wish someone from asus could let us know if this is the way to go or if there are any other settings to get a 4GPU system running with a 950 pro drive. It may be just a tiny BIOS bug as on every 10th boot it worked fine. This board is the first choice of many people that want to build a 4 GPU system. To put a 950 or 960 in there is a no brainer. I bet a lot of people will face this problem. It took a long time to find a solutionon my own. There should be no need for someone to go through this.

[slp_pleh]

CrusherXL3D wrote:
I would wish someone from asus could let us know if this is the way to go or if there are any other settings to get a 4GPU system running with a 950 pro drive. It may be just a tiny BIOS bug as on every 10th boot it worked fine. This board is the first choice of many people that want to build a 4 GPU system. To put a 950 or 960 in there is a no brainer. I bet a lot of people will face this problem. It took a long time to find a solutionon my own. There should be no need for someone to go through this.

I wonder if the performance difference is because the M.2 card slot is only getting half the pcie lanes, vs the U.2 getting more? Maybe you can tell from the BIOS status of the card? I don't know how to check how many lanes a m.2 card is using though..

I called them again today (another hour wasted on the phone) and asked them to re-escalate the case to their engineering team. I'm making a big complaint about the problem I am having and have asked their 4th level (manager in Canada) for a call back to discuss the huge pain this has been for me trying to get support--so maybe that will help us get this fixed.

[Brighttail]

Well there is one area in the ADVANCED area of the BIOS you can set your 4-way SLI to x8x8x8x8 which may allow the m.2 to be functional then. Beyond that I have no other answers. ON the R5E I used to be able to run two GPUS at x16/x16 in SLI AND have two m.2 drives running at x4/x4. ON this board it won't handle that. I ended up with x16/x8 on the GPUS and x4/x4 on the m.2s.

[slp_pleh]

Thanks--I do remember trying that option, or at least noticing it was set to all x8 but sadly no luck.

I finally got asus tech support to open an engineering ticket to look into it further, hopefully its a setting I haven't found or an easy fix.

[WestManiaC]

slp_pleh wrote:
Thanks--I do remember trying that option, or at least noticing it was set to all x8 but sadly no luck.

I finally got asus tech support to open an engineering ticket to look into it further, hopefully its a setting I haven't found or an easy fix.

says so for rampage v 10 edition on their site

*Enjoy 4-way SLI plus M.2 and U.2 at the same time

[slp_pleh]

WestManiaC wrote:
says so for rampage v 10 edition on their site

*Enjoy 4-way SLI plus M.2 and U.2 at the same time

Thanks for pointing that out, it is very helpful to know it's documented !

[CrusherXL3D]

Hi!

I have exactly the same problem! 4 GPUs installed, Samsung Evo 950 NVMe drive with WIn10 installed and I am not able to boot. It just stucks at the loading dots from win10. The SSD LED is blinking quickly and then stopps at one point. Thats the moment when the system freezes.

The weirdest things is that every 10 boots or so, it actually works! I have all 4 GPUs available for rendering and full speed of the M2 950NVMe drive! As soon as i warm reboot, cold reboot or just reset, the stystem does not start anymore. As soon as I remove the 4th GPU from the last slot, the system boots just fine. It must have something to to with the lanes sharing. I have a 40 lanes CPU and the GPUs are running at 8x. So that makes actually 8x4 = 32 lanes. Plus the 4 lanes for the M2 brings me to 36 used lanes. So what is causing the problem? I disabled EVERY other harware on the board to be sure those lanes are free.

I have tried all possible combinations in the advanced section in the Bios (1502) like switching off all Components like USB 3.1, AUdiocontroller etc. I have unplugged all USB3 oboard connectors trying to solve the issue. No sucess.


I really really wonder if anybody has build a 4 GPU system with that board and an M2 drive. To avoid any comments about the gaming issues with 4way SLI: I build this system purely for rendering purposes. So i just need the GPUs accessible without any SLI bridges etc....

There must be a bug in the BIOS. I have tried all available version an all of thm have the same behavior. Every x boots, the system works properly. In between there is nothing you can do to get it up an running.

My last two ideas are: Change the Evo 950 to a 850. The idea is that the 950 is a x4 card, while the 850 is just an 2x. So those two additional lanes may help. The other idea i will try is to get an external enclosure with a U2 port for the 950. The manual explicity said that the U2 port supports NVMe drives. There is nothing stated about those drives in the M2 port.

Please keep that post updates as I can imagine a lot of people will run into this issue. I spend my whole weekend with investigation without any sucess... Very very disapoin

[CrusherXL3D]

Update: The U2 port idea worked with 4GPUs! And even more: Its way more fast than on the internal M2 slot. 2500 read/s and 1500 write/s

So if you have 4GPUs and the Evo 950 NVMe you need to get an U2 cable (Funtin U.2 (SFF-8639) to HD Mini-SAS (SFF-8643) Cable for 2.5" NVMe SSD) and a tray (Ableconn IU2-M2132 M.2 NVMe SSD to U.2 (SFF-8639)).

With that you can connect the Evo to the U2 port, what really supports NVMe drives.

[slp_pleh]

CrusherXL3D wrote:
Update: The U2 port idea worked with 4GPUs! And even more: Its way more fast than on the internal M2 slot. 2500 read/s and 1500 write/s

So if you have 4GPUs and the Evo 950 NVMe you need to get an U2 cable (Funtin U.2 (SFF-8639) to HD Mini-SAS (SFF-8643) Cable for 2.5" NVMe SSD) and a tray (Ableconn IU2-M2132 M.2 NVMe SSD to U.2 (SFF-8639)).

With that you can connect the Evo to the U2 port, what really supports NVMe drives.

That is good to know that the external port makes it work. Thanks for the info! It does seem like a BIOS problem. I'll call them again and bug them. You should call them as well so we can maybe get a quicker fix if they realize that it is their issue.

Side note/rant: I have been wasting hours of my time with Asus tech support. They seem to have no idea why this isn't working and keep trying to get me to test things that won't fix it. (install an OS first and then call us back), or can you try it with BIOS setting X and then call us back. I made them hold while I tried it because calling in again will be another 20-30 min of hold time, and then you have to completely re-explain everything. So frustrating, but I guess this is the only path to get them to make a bug fix.

[Brighttail]

slp_pleh wrote:
That is good to know that the external port makes it work. Thanks for the info! It does seem like a BIOS problem. I'll call them again and bug them. You should call them as well so we can maybe get a quicker fix if they realize that it is their issue.

Side note/rant: I have been wasting hours of my time with Asus tech support. They seem to have no idea why this isn't working and keep trying to get me to test things that won't fix it. (install an OS first and then call us back), or can you try it with BIOS setting X and then call us back. I made them hold while I tried it because calling in again will be another 20-30 min of hold time, and then you have to completely re-explain everything. So frustrating, but I guess this is the only path to get them to make a bug fix.

In this situation, the first line techs won't be able to give you much. This is something that has to do with the BIOS and potentially the actual chipset and how it handles the PCI-e Lanes.

You'd probably get quicker information having started a thread asking Raja for help. He once posted a schematic of how the R5E10 handles PCI-e lanes to show why I couldn't have x16 /x4 / x16 / x4 even with a 40 lane CPU on this board when it worked with the R5E. Unfortunately for me it only will work x16 / x4 / x8 / x4 😞