Z790-E Gaming Wi-Fi & Nvme support

[Bfxt]

I’ve been studying the specs of this motherboard, along with the bifurcation faq at https://www.asus.com/support/faq/1037507/

I’ve yet to find a block diagram of the architecture of this board.  

Seemingly one could support eight simultaneous pcie4x4 nvme devices running at their full pcie4x4 speeds.
Five in the on-board m2 slots, and three more in the three on-board pcie slots using three low cost passive pcie-to-single-m2 adapter cards. (This assumes no other use of the pcie slots, e.g. no graphics card.)

These eight Nvme sticks would be attached via some combination of cpu-attached and pch-attached pcie lanes.

Questions:

1)  Is the above correct? 

2A)  Could all eight Nvme devices theoretically be read from at full speed simultaneously?
2B)  If not, would that be due to the pch having to multiplex amongst the pch-attached Nvme devices to route them to the cpu?

3)  Does the motherboard incorprate any additional pcie switch(es), in order to accommodate all these pcie4 lanes?

I’m asking because this is one of the only consumer boards I’ve found that appears to support eight Nvme devices at pcie4x4. The others are also asus.  I’m wondering why that is, or is there something I’m not understanding.  Other vendors’ offerings seem to be limited to either fewer, or slower, lanes.

Thanks in advance.

[Jiaszzz_ROG]

Hello, @Bfxt 

In the case of the ROG STRIX Z790-E GAMING WIFI, according to the table, Hyper M.2 series Cards need to be installed in the PCIE16(G5)_1 slot.
Hyper M.2 supports either one SSD operating at X8 or X16, or two SSDs both operating at X8.

When installing one SSD, the slot positions on Hyper M.2:

When installing two SSDs, the slot positions on Hyper M.2:

As per the remark, when installing Hyper M.2 with two SSDs, M.2_1 will disconnect because they share bandwidth.
And when Hyper M.2 is installed with two SSDs, BIOS settings should be configured to [X8/X8].

According to specifications, M.2_2 through M.2_5, plus the two SSDs on Hyper M.2, allow for a maximum of six SSDs to be installed on the motherboard.
Regarding question three, could you please provide further contextual details?

Thank you.

[Bfxt]

Thank you very much for replying! For future readers, here are my findings. I'm no expert and all input is welcome!

>> 1) Is the above correct?

    Yes.

>> 2A) Could all eight Nvme devices theoretically be read from at full speed simultaneously?

    No.

>> 2B) If not, would that be due to the pch having to multiplex amongst the pch-attached Nvme devices to route them to/from the cpu?

    Yes.

>> 3) Does the motherboard incorporate any additional pcie switch(es), in order to accommodate all these pcie4 lanes?

     No.

Seems like Asus made a couple design decisions with this board that fit my use-case pretty well:

•  Limiting to four Sata3 ports frees up an extra pcie4x4 for an additional m2 nvme slot.
•  Inclusion of the M2_1 pcie5x4 slot that shares bifurcated bandwidth with the main PCIE16(G5)slot obviates the need for an M2 Hypercard.

I didn't know about the bios bifurcation setting. I think Auto is doing the right thing: Slot PCIEX16(G5) defaults to gen5x16 mode. However if an nvme device is sensed as being installed in slot M2_1, then PCIEX16(G5) automatically “bifurcates” into gen5x8, and M2_1 simultaneously hosts gen5x4. (Theoretically there's enough cpu bandwidth available for M2_1 to support gen5x8 but, best I can tell, the current m2 spec maxes out at x4.)

With this motherboard, the example of a Hypercard hosting two nvme sticks can instead be accomplished with:

• One nvme stick in M2_1, and
• A second nvme stick in PCIEX16(G5), via a cheapo pcie-to-single-m2 passive carrier card.

I have confirmed that the board supports eight simultaneous nvme sticks, each maxing out at either gen4x4 or gen5x4 depending on the slot (and the nvme device of course).

I've annotated the following diagram with what each of the eight slots can simultaneously host. (Doh I missed transcribing the _1 and _2 for the two yellow pcie slots)

There are three total cpu-attached slots (green is gen5 and blue is gen4). The remaining five are pch-attached (yellow, all gen4). (PCH=Platform Controller Hub, provided by the chipset, which in this case is the Intel Z790.)

These five gen4x4 pch-attached slots are multiplexed by the chipset into two gen4x4 "dmi" connections between the chipset and the cpu (not shown above, but see links below). So all twenty yellow-slot lanes are potentially competing for the same eight lanes of dmi bandwidth. It's unfortunate that intel’s dmi connection totals gen4x8, instead of, say, gen5x8 or gen4x16.

Cpu/chipset/lanes architectural block diagrams:

https://www.intel.com/content/www/us/en/products/docs/chipsets/desktop-chipsets/z790-chipset-brief.h...

Top left of https://tpucdn.com/cpu-specs/images/connectivity/intel-lga-1700.png

The lspci command shows system pcie details: https://www.baeldung.com/linux/lspci

[achugh]

Hi @Jiaszzz_ROG , please see my post at https://rog-forum.asus.com/t5/gaming-motherboards/crosshair-viii-formula-with-hypercard-in-pcie-16-1... where I documented that a SINGLE HYPER M.2 Card as you have shown here in your post CANNOT accept 2 NVMe drives on these Z690 or Z790 chipset boards. Or any consumer DIY motherboard.

Has ASUS development and design team tested what you have posted here because as I mentioned in my post, I had to return the HYPER M.2 card as it would not recognize the 2nd NVMe on Z790 DARK HERO board.

[Jiaszzz_ROG]

Hello, @Bfxt 

Upon verification, please refer to the "Connectors with shared bandwidth" section in the ROG MAXIMUS Z790 DARK HERO manual:
- When the Hyper M.2 card is installed in the PCIEX16 (G5) slot without installing a GPU, the motherboard can recognize two SSDs by setting the PCIe Bandwidth Bifurcation Configuration to [X8/X8]. 

- When the Hyper M.2 card is installed in the PCIEX16(G5) slot along with a GPU in another PCIEX16(G5) slot, the motherboard can only recognize one SSD.
This limitation is due to shared bandwidth, and the M.2_3, M.2_4, and M.2_5 slots, which utilize bandwidth provided by the Z790 chipset, are not affected by the CPU bandwidth.

Thank you.

[achugh]

Hi @Jiaszzz_ROG , thank you for responding back here.

I would like to state that the very first picture you shared from the Bifurcation FAQ table does not list Z790 DARK HERO board. This was the first problem I faced 3 months ago when I was working with customer support as they first told me that Z790 DARK HERO board does not even support HYPER M.2 card as it is not event listed on this table (I hope your team can request this table to be updated to avoid these issues for other customers like me). After a little bit of back and forth when I started to understand things on my own by running multiple trial and error tests I concluded my results in the post I shared that these HYPER M.2 card cannot split PCIe lanes.

If you look at the LINE ABOVE RED LINES you highlighted in your picture; the 4th column reads the following for ROG MAXIMUS 790 HERO board 1(X16)/ 2(X8 + X8). This information is under the column header M.2 SSD Quantity/Speed.

The way I read this information and I was able to prove it when I had the card is as follows:

NOTE: The following interpretation assumes there are NO M.2 Drives installed in the shared M.2_1 slot which shares its bandwidth with PCIe16(G5)_2.

When the HYPER M.2 card is installed in the PCIe16(G5)_1 first slot near the CPU and there is nothing installed in 2nd PCIe16(G5)_2 >>> the PCIe16(G5)_1 is now going to run at X16 SPEED. At this speed only 1 M.2 Drive will be detected which is shown by the very first 1(X16) as the use case or option before the SLASH (/) in this row. This is because this card cannot split PCIe lanes as I explain in my other post.

When TWO HYPER M.2 card are installed i.e. first card in the PCIe16(G5)_1 first slot near the CPU and 2nd card in the PCIe16(G5)_2 >>> both PCIe16(G5)_1 and PCIe16(G5)_2 are now going to run at X8 SPEED. In this case both cards will detect only 1 M.2 Drive which is shown by the 2(X8 + X8) as the 2nd use case or option AFTER the SLASH (/) in this row. We get TWO M.2 Drives because each card gives us 1 Drive as they again cannot split PCIe lanes.

NOW IF WE LOOK AT THE NEXT ROW IN YOUR PICTURE, it reads 1(X8) /X0

In this case, if any card is installed in PCIe16(G5)_1 say a graphics card which is a typical use case, then PCIe16(G5)_2 will only get X8 lanes and this card will support 1 M.2 Drive which is shown by 1(X8) again. Because the card cannot split PCIe lanes.

In the last use case if anything is inserted into M.2_1 slot then PCIe16(G5)_2 gets no lanes as it shares it with M.2_1 and hence nothing will work as M.2_1 is give priority by default which is depicted by X0 in the table.

If I misunderstood reading this table, I would be happy to get myself corrected and in fact go back and get this HYPER M.2 Card again.

Everything I shared here is exactly how my setup worked when I owned this card and I spent close to 25 days trying to make it work before I returned it since it was not giving me more than 1 M.2 Drive which I could get with M.2_1 slot anyway so I was not getting any benefit of owning this card.

My only intent or hope is for ASUS team make things clear for consumers as it will make the consumers happy. Happy consumers means less customer support and more profits and benefits to the ASUS company that we all love so much to flourish. My 2nd intent is to encourage ASUS engineering team to actually create a newer version of these cards that can split the PCIe lanes so we all can get the benefit as things will work the way we all assume before delving into technical details as to why they are not working as these technical details are not clearly and easily available to us consumers.

[Jiaszzz_ROG]

Hello, @achugh 

Thank you for sharing related details.
After verification, the [Motherboard] Compatibility of PCIE Bifurcation between Hyper M.2 Series Cards and Add-On Graphic Cards does not include the ROG MAXIMUS Z790 DARK HERO because, although it can connect to Hyper M.2, it can only recognize one M.2 SSD at a time.
This means that regardless of which slot the Hyper M.2 is installed in, only one M.2 SSD can be connected to the Hyper M.2.

Thank you.

[achugh]

Hi @Jiaszzz_ROG thank you for confirming that Z79 DARK HERO will take only 1 drive.

Just for my own understanding using the above FAQ table, how should Z790 HERO (1st ROW) be interpreted where it says 2(X8 + X8)?

The way I read this part is that Z790 HERO will support up to a maximum of 2 NVMe drives in the X8 + X8 configuration but this requires the use of 2 HYPER M.2 Card i.e. 1st HYPER M.2 card in PCIEX15(G5)_1 and 2nd HYPER M.2 Card in PCIEX16(G5)_2 slot. In other words, the HYPER M.2 card will only support 1 NVMe drive at a time.

If my understanding is correct then this is what I am very humbly try to pass to ASUS team to update and understand here. The pictures showing 2 NVMe Drives in this HYPER M.2 Card should be removed from the FAQ or should have a note as to which Intel CPUs will support them. The LGA1700 socket CPUs are not supporting more than 1 NVMe drive in HYPER M.2 card is what I am trying to convey here.

[Jiaszzz_ROG]

Hello, @achugh 

For installing the Hyper M.2 on the first model, the ROG MAXIMUS Z790 HERO, please refer to my earlier response provided to the Bfxt user.
Additionally, please note that the "ROG MAXIMUS Z790 DARK HERO" does not support Hyper M.2, while the ROG MAXIMUS Z790 HERO does. These are different models within the same series.

Thank you.

[achugh]

Thank you @Jiaszzz_ROG for responding to me. We can end this conversation here because you have confirmed for my Z790 DARK HERO board the HYPER M.2 Card is not supported and I can also confirm that it only supports just 1 M.2 NVMe drive in this card which is something you have also confirmed.

Thanks again for your help.