©ASUSTEK COMPUTER INC. ALL RIGHTS RESERVED.

estatemagnate · ‎09-29-2023

Hi All,
I hope you are doing well.
I have ROG Strix X570-E WIFI with AMD Ryzen 9 5950X
I fitted this disk:
Crucial P5 Plus 2TB PCIe M.2 2280SS Gaming SSD
https://uk.crucial.com/ssd/p5-plus/CT1000P5PSSD5
It supposed to work in Gen4 however in the BIOS it is detected on Auto but if I change to Gen4 this disk disappears.

Is there anything else in the BIOS that I should change to make it work in Gen4 and/or verify in the system if it's Gen4?

fio seq write tests give me about 4000MB/s which is kind of above Gen3 spec but if so with fio I should get around 6000MB/s.
Please advise.
Best regards

estatemagnate · ‎09-29-2023

Just to clarify the BIOS has been upgraded to the latest and the actual speed of the disk fio has calculates is slightly under 3GB/s so it's clearly Gen3 speed.

ReatKay · ‎09-29-2023

That the NVMe is disappearing is quite weird, which slot did you put it on and are your other M.2 slots also populated?

First of all, you name seq write speeds of 6'000 MB/s.

Crucial lists the P5 Plus:

Sequential Write

5,000 MB/s

(this always means: Up to 5'000 MB/s - reachable under absolute perfect system & test conditions, but even if you replicate those, it might never get to this value - and due to factors like error margins, differences in each manufactured piece going into sale etc.)

Just making clear: You won't even see those 5'000MB/s write transfer rate, the max you can expect (according to test coming from a large german IT magazine) is 4'700MB/s - but just as peak - average will be like around 4'300MB/s write transfer rate.

Yes, it's a shame - manufacturers still list speeds achieved under perfect lab environment, taking the best results from a batch of tests. A simple "speed UP TO" is enough, to make it legally correct. Based on your supplied Write Transfer Speeds,I just checked a test of the P5 Plus. Sustained Write Speeds during testing have been 4300 MB/s (average read transfer) and 4700 MB/s (absolute peak value), with a lower write speed of 4'100 as soon as the P5's cache is full. The 4'000 MB/s are so close to the 4'100 sustained write speed after cache the test's speaking of, that I can't see any issue for the write speed. PS: Expect even less, when the last "cache" is exhausted, then you won't get more than like 1700 MB/s, but thats after writing more like 200GB in a run.

Under perfect lab environment with probably an optimized test, they might actually have reached that. But such values always include an error margin, so the test speaking of a peak of 4'700MB - that would legally make the 5'000 MB/s from Crucial's own listing "legit" - not right, but legally acceptable. Taking your 4'000 and relating it to the 4'300 from the test, that makes a difference of 200MB/s - which is - still within "acceptable" error margin. When I benchmark my drives, the differences are sometimes even more.

If you tell me your read transfer speeds, then we can compare those aswel - but simply looking at the advertised, test values and your own test values of write speed, that looks all still plausible to me not really indicating a false generation problem. Max advertised read write speed of the P5 Plus is 6'600 MB/s - but again, advertised lab test speeds, afaik, during test the average value was around 5'600 MB/s

The hard and sad truth is: You just bought - for the current time - a performance-wise not so great NVMe, whose manufacturer also exaragated its performance quite heavier than what I'm used to on my drives. My retailer doesn't recommend the Plus versions of the Crucial P5 for those seeking real performance, as it doesn't fully deliver what it promises - but marks them as still "performant enough" for gaming, the OS, 0815 usage.

Here the test averages: (Source: German magazine Chip

Average Read Speed

Average Write Speed

estatemagnate · ‎09-29-2023

Hi,
Thank you for your detailed insight.
I agree these are low end consumer NVMes so I was kind of expecting that the figures they provide is marketing.
I have two of these in both M.2 slots. I am testing on the one that is not busy.
The tested one is in M.2_2 slot below the GPU not under the CPU.
I have seen a video on YT where a man was explaining the difference between NVMe going through PCH and CPU.
Apparently if it goes through the chipset it is slower and if possible should be changed in the BIOS.

I tested Micron 7400 MAX and this one will give you 6.9GB/s seqwrite sustained but it's over 3 times more expensive.

This is a complete read sequential write test. I used 4MB block size which is optimal for these:

fio --name=seqwrite --rw=write --direct=1 --ioengine=libaio --bs=4M --numjobs=4 --size=30G --runtime=600
seqwrite: (g=0): rw=write, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=1
...
fio-3.35
Starting 4 processes
seqwrite: Laying out IO file (1 file / 30720MiB)
seqwrite: Laying out IO file (1 file / 30720MiB)
seqwrite: Laying out IO file (1 file / 30720MiB)
seqwrite: Laying out IO file (1 file / 30720MiB)
Jobs: 4 (f=4): [W(4)][100.0%][w=1640MiB/s][w=410 IOPS][eta 00m:00s]
seqwrite: (groupid=0, jobs=1): err= 0: pid=141705: Fri Sep 29 17:37:27 2023
  write: IOPS=178, BW=715MiB/s (750MB/s)(30.0GiB/42940msec); 0 zone resets
    slat (usec): min=131, max=3651, avg=227.28, stdev=67.99
    clat (usec): min=1653, max=126773, avg=5362.64, stdev=10067.31
     lat (usec): min=1841, max=127083, avg=5589.92, stdev=10074.48
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    9], 95.00th=[   11],
     | 99.00th=[   83], 99.50th=[   88], 99.90th=[   95], 99.95th=[  100],
     | 99.99th=[  127]
   bw (  KiB/s): min=40960, max=1228800, per=25.11%, avg=735930.73, stdev=432686.39, samples=85
   iops        : min=   10, max=  300, avg=179.67, stdev=105.64, samples=85
  lat (msec)   : 2=0.01%, 4=63.59%, 10=31.43%, 20=3.41%, 50=0.13%
  lat (msec)   : 100=1.39%, 250=0.03%
  cpu          : usr=0.77%, sys=3.37%, ctx=7704, majf=0, minf=13
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7680,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=141706: Fri Sep 29 17:37:27 2023
  write: IOPS=178, BW=716MiB/s (751MB/s)(30.0GiB/42921msec); 0 zone resets
    slat (usec): min=114, max=514, avg=199.50, stdev=39.44
    clat (usec): min=988, max=136316, avg=5387.89, stdev=10039.36
     lat (usec): min=1180, max=136492, avg=5587.39, stdev=10045.67
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    9], 95.00th=[   11],
     | 99.00th=[   83], 99.50th=[   87], 99.90th=[   93], 99.95th=[   99],
     | 99.99th=[  138]
   bw (  KiB/s): min=40960, max=1245184, per=25.13%, avg=736316.24, stdev=432785.28, samples=85
   iops        : min=   10, max=  304, avg=179.76, stdev=105.66, samples=85
  lat (usec)   : 1000=0.01%
  lat (msec)   : 2=0.01%, 4=63.44%, 10=31.60%, 20=3.37%, 50=0.14%
  lat (msec)   : 100=1.38%, 250=0.04%
  cpu          : usr=0.68%, sys=2.59%, ctx=7709, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7680,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=141707: Fri Sep 29 17:37:27 2023
  write: IOPS=178, BW=715MiB/s (750MB/s)(30.0GiB/42938msec); 0 zone resets
    slat (usec): min=134, max=65560, avg=242.90, stdev=752.92
    clat (usec): min=1507, max=101835, avg=5346.76, stdev=9946.79
     lat (usec): min=1733, max=127677, avg=5589.67, stdev=10033.64
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    9], 95.00th=[   10],
     | 99.00th=[   83], 99.50th=[   88], 99.90th=[   93], 99.95th=[   96],
     | 99.99th=[  103]
   bw (  KiB/s): min=40960, max=1236992, per=25.12%, avg=736027.11, stdev=432552.79, samples=85
   iops        : min=   10, max=  302, avg=179.69, stdev=105.60, samples=85
  lat (msec)   : 2=0.03%, 4=63.48%, 10=31.63%, 20=3.29%, 50=0.14%
  lat (msec)   : 100=1.42%, 250=0.01%
  cpu          : usr=0.76%, sys=3.47%, ctx=7698, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7680,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1
seqwrite: (groupid=0, jobs=1): err= 0: pid=141708: Fri Sep 29 17:37:27 2023
  write: IOPS=178, BW=716MiB/s (750MB/s)(30.0GiB/42929msec); 0 zone resets
    slat (usec): min=124, max=615, avg=211.10, stdev=41.00
    clat (usec): min=1691, max=115511, avg=5377.46, stdev=10061.30
     lat (usec): min=1915, max=115756, avg=5588.57, stdev=10069.35
    clat percentiles (msec):
     |  1.00th=[    3],  5.00th=[    3], 10.00th=[    3], 20.00th=[    3],
     | 30.00th=[    3], 40.00th=[    3], 50.00th=[    3], 60.00th=[    3],
     | 70.00th=[    5], 80.00th=[    5], 90.00th=[    9], 95.00th=[   10],
     | 99.00th=[   83], 99.50th=[   88], 99.90th=[   97], 99.95th=[  103],
     | 99.99th=[  116]
   bw (  KiB/s): min=40960, max=1236992, per=25.12%, avg=736219.86, stdev=432822.89, samples=85
   iops        : min=   10, max=  302, avg=179.74, stdev=105.67, samples=85
  lat (msec)   : 2=0.03%, 4=63.58%, 10=31.46%, 20=3.41%, 50=0.10%
  lat (msec)   : 100=1.37%, 250=0.05%
  cpu          : usr=0.72%, sys=3.09%, ctx=7693, majf=0, minf=12
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,7680,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=2862MiB/s (3001MB/s), 715MiB/s-716MiB/s (750MB/s-751MB/s), io=120GiB (129GB), run=42921-42940msec

Disk stats (read/write):
  nvme1n1: ios=0/1004732, merge=0/9798, ticks=0/4736764, in_queue=4736860, util=99.82%

and read test:

fio --filename=seqwrite.0.0 --direct=1 --rw=read --bs=4M --ioengine=libaio --iodepth=256 --runtime=120 --numjobs=4 --time_based --group_reporting --name=iops-test-job --eta-newline=1 --readonly
iops-test-job: (g=0): rw=read, bs=(R) 4096KiB-4096KiB, (W) 4096KiB-4096KiB, (T) 4096KiB-4096KiB, ioengine=libaio, iodepth=256
...
fio-3.35
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=2026MiB/s][r=506 IOPS][eta 01m:57s]
Jobs: 4 (f=4): [R(4)][3.3%][r=3844MiB/s][r=961 IOPS][eta 01m:56s]
Jobs: 4 (f=4): [R(4)][4.2%][r=3473MiB/s][r=868 IOPS][eta 01m:55s]
Jobs: 4 (f=4): [R(4)][5.0%][r=3461MiB/s][r=865 IOPS][eta 01m:54s]
Jobs: 4 (f=4): [R(4)][6.7%][r=3364MiB/s][r=841 IOPS][eta 01m:52s] 
Jobs: 4 (f=4): [R(4)][7.5%][r=3189MiB/s][r=797 IOPS][eta 01m:51s]
Jobs: 4 (f=4): [R(4)][9.2%][r=3429MiB/s][r=857 IOPS][eta 01m:49s] 
Jobs: 4 (f=4): [R(4)][10.0%][r=2861MiB/s][r=715 IOPS][eta 01m:48s]
Jobs: 4 (f=4): [R(4)][10.8%][r=3405MiB/s][r=851 IOPS][eta 01m:47s]
Jobs: 4 (f=4): [R(4)][11.7%][r=3193MiB/s][r=798 IOPS][eta 01m:46s]
Jobs: 4 (f=4): [R(4)][13.3%][r=3457MiB/s][r=864 IOPS][eta 01m:44s] 
Jobs: 4 (f=4): [R(4)][15.0%][r=3560MiB/s][r=890 IOPS][eta 01m:42s] 
Jobs: 4 (f=4): [R(4)][16.7%][r=3325MiB/s][r=831 IOPS][eta 01m:40s] 
Jobs: 4 (f=4): [R(4)][17.5%][r=2805MiB/s][r=701 IOPS][eta 01m:39s]
Jobs: 4 (f=4): [R(4)][18.3%][r=3500MiB/s][r=875 IOPS][eta 01m:38s]
Jobs: 4 (f=4): [R(4)][19.3%][r=2957MiB/s][r=739 IOPS][eta 01m:36s]
Jobs: 4 (f=4): [R(4)][20.8%][r=3497MiB/s][r=874 IOPS][eta 01m:35s] 
Jobs: 4 (f=4): [R(4)][21.7%][r=3017MiB/s][r=754 IOPS][eta 01m:34s]
Jobs: 4 (f=4): [R(4)][22.5%][r=3113MiB/s][r=778 IOPS][eta 01m:33s]
Jobs: 4 (f=4): [R(4)][23.3%][r=3033MiB/s][r=758 IOPS][eta 01m:32s]
Jobs: 4 (f=4): [R(4)][25.0%][r=2877MiB/s][r=719 IOPS][eta 01m:30s] 
Jobs: 4 (f=4): [R(4)][26.7%][r=3220MiB/s][r=805 IOPS][eta 01m:28s] 
Jobs: 4 (f=4): [R(4)][28.3%][r=2976MiB/s][r=744 IOPS][eta 01m:26s] 
Jobs: 4 (f=4): [R(4)][30.0%][r=3396MiB/s][r=849 IOPS][eta 01m:24s] 
Jobs: 4 (f=4): [R(4)][30.8%][r=2925MiB/s][r=731 IOPS][eta 01m:23s]
Jobs: 4 (f=4): [R(4)][31.7%][r=2517MiB/s][r=629 IOPS][eta 01m:22s]
Jobs: 4 (f=4): [R(4)][32.5%][r=3137MiB/s][r=784 IOPS][eta 01m:21s]
Jobs: 4 (f=4): [R(4)][34.2%][r=2766MiB/s][r=691 IOPS][eta 01m:19s] 
Jobs: 4 (f=4): [R(4)][35.8%][r=2917MiB/s][r=729 IOPS][eta 01m:17s] 
Jobs: 4 (f=4): [R(4)][36.7%][r=3540MiB/s][r=885 IOPS][eta 01m:16s]
Jobs: 4 (f=4): [R(4)][37.5%][r=3277MiB/s][r=819 IOPS][eta 01m:15s]
Jobs: 4 (f=4): [R(4)][38.7%][r=3181MiB/s][r=795 IOPS][eta 01m:13s]
Jobs: 4 (f=4): [R(4)][39.2%][r=2945MiB/s][r=736 IOPS][eta 01m:13s]
Jobs: 4 (f=4): [R(4)][40.8%][r=2996MiB/s][r=749 IOPS][eta 01m:11s] 
Jobs: 4 (f=4): [R(4)][42.5%][r=3089MiB/s][r=772 IOPS][eta 01m:09s] 
Jobs: 4 (f=4): [R(4)][43.3%][r=2817MiB/s][r=704 IOPS][eta 01m:08s]
Jobs: 4 (f=4): [R(4)][45.0%][r=3269MiB/s][r=817 IOPS][eta 01m:06s] 
Jobs: 4 (f=4): [R(4)][45.8%][r=3500MiB/s][r=875 IOPS][eta 01m:05s]
Jobs: 4 (f=4): [R(4)][47.5%][r=2893MiB/s][r=723 IOPS][eta 01m:03s] 
Jobs: 4 (f=4): [R(4)][48.3%][r=3389MiB/s][r=847 IOPS][eta 01m:02s]
Jobs: 4 (f=4): [R(4)][50.0%][r=3345MiB/s][r=836 IOPS][eta 01m:00s] 
Jobs: 4 (f=4): [R(4)][52.1%][r=1100MiB/s][r=275 IOPS][eta 00m:57s] 
Jobs: 4 (f=4): [R(4)][53.3%][r=1228MiB/s][r=307 IOPS][eta 00m:56s]
Jobs: 4 (f=4): [R(4)][55.5%][r=788MiB/s][r=197 IOPS][eta 00m:53s]  
Jobs: 4 (f=4): [R(4)][55.8%][r=991MiB/s][r=247 IOPS][eta 00m:53s]
Jobs: 4 (f=4): [R(4)][57.5%][r=1259MiB/s][r=314 IOPS][eta 00m:51s] 
Jobs: 4 (f=4): [R(4)][59.2%][r=1164MiB/s][r=291 IOPS][eta 00m:49s] 
Jobs: 4 (f=4): [R(4)][60.0%][r=1067MiB/s][r=266 IOPS][eta 00m:48s]
Jobs: 4 (f=4): [R(4)][60.8%][r=1039MiB/s][r=259 IOPS][eta 00m:47s]
Jobs: 4 (f=4): [R(4)][61.7%][r=1151MiB/s][r=287 IOPS][eta 00m:46s]
Jobs: 4 (f=4): [R(4)][62.5%][r=967MiB/s][r=241 IOPS][eta 00m:45s]
Jobs: 4 (f=4): [R(4)][64.7%][r=1016MiB/s][r=254 IOPS][eta 00m:42s] 
Jobs: 4 (f=4): [R(4)][65.0%][r=1355MiB/s][r=338 IOPS][eta 00m:42s]
Jobs: 4 (f=4): [R(4)][66.7%][r=1140MiB/s][r=285 IOPS][eta 00m:40s] 
Jobs: 4 (f=4): [R(4)][68.3%][r=1108MiB/s][r=277 IOPS][eta 00m:38s] 
Jobs: 4 (f=4): [R(4)][69.2%][r=1027MiB/s][r=256 IOPS][eta 00m:37s]
Jobs: 4 (f=4): [R(4)][70.0%][r=1259MiB/s][r=314 IOPS][eta 00m:36s]
Jobs: 4 (f=4): [R(4)][70.8%][r=1079MiB/s][r=269 IOPS][eta 00m:35s]
Jobs: 4 (f=4): [R(4)][71.7%][r=1055MiB/s][r=263 IOPS][eta 00m:34s]
Jobs: 4 (f=4): [R(4)][73.3%][r=1316MiB/s][r=329 IOPS][eta 00m:32s] 
Jobs: 4 (f=4): [R(4)][75.0%][r=924MiB/s][r=231 IOPS][eta 00m:30s]  
Jobs: 4 (f=4): [R(4)][77.3%][r=1152MiB/s][r=288 IOPS][eta 00m:27s]
Jobs: 4 (f=4): [R(4)][78.3%][r=944MiB/s][r=236 IOPS][eta 00m:26s]  
Jobs: 4 (f=4): [R(4)][79.2%][r=1051MiB/s][r=262 IOPS][eta 00m:25s]
Jobs: 4 (f=4): [R(4)][80.0%][r=1399MiB/s][r=349 IOPS][eta 00m:24s]
Jobs: 4 (f=4): [R(4)][81.7%][r=992MiB/s][r=248 IOPS][eta 00m:22s]  
Jobs: 4 (f=4): [R(4)][83.3%][r=988MiB/s][r=247 IOPS][eta 00m:20s]  
Jobs: 4 (f=4): [R(4)][85.0%][r=940MiB/s][r=235 IOPS][eta 00m:18s]  
Jobs: 4 (f=4): [R(4)][86.7%][r=992MiB/s][r=248 IOPS][eta 00m:16s]  
Jobs: 4 (f=4): [R(4)][88.3%][r=1163MiB/s][r=290 IOPS][eta 00m:14s] 
Jobs: 4 (f=4): [R(4)][89.2%][r=2925MiB/s][r=731 IOPS][eta 00m:13s]
Jobs: 4 (f=4): [R(4)][90.0%][r=2953MiB/s][r=738 IOPS][eta 00m:12s]
Jobs: 4 (f=4): [R(4)][90.8%][r=2969MiB/s][r=742 IOPS][eta 00m:11s]
Jobs: 4 (f=4): [R(4)][92.5%][r=2796MiB/s][r=699 IOPS][eta 00m:09s] 
Jobs: 4 (f=4): [R(4)][93.3%][r=3612MiB/s][r=903 IOPS][eta 00m:08s]
Jobs: 4 (f=4): [R(4)][94.2%][r=2813MiB/s][r=703 IOPS][eta 00m:07s]
Jobs: 4 (f=4): [R(4)][95.0%][r=3297MiB/s][r=824 IOPS][eta 00m:06s]
Jobs: 4 (f=4): [R(4)][95.8%][r=3029MiB/s][r=757 IOPS][eta 00m:05s]
Jobs: 4 (f=4): [R(4)][97.5%][r=3720MiB/s][r=930 IOPS][eta 00m:03s] 
Jobs: 4 (f=4): [R(4)][98.3%][r=3133MiB/s][r=783 IOPS][eta 00m:02s]
Jobs: 4 (f=4): [R(4)][100.0%][r=3212MiB/s][r=803 IOPS][eta 00m:00s]
iops-test-job: (groupid=0, jobs=4): err= 0: pid=185865: Sat Sep 30 01:41:45 2023
  read: IOPS=593, BW=2373MiB/s (2488MB/s)(278GiB/120138msec)
    slat (usec): min=65, max=145221, avg=6733.14, stdev=6306.22
    clat (msec): min=132, max=5906, avg=1707.34, stdev=976.12
     lat (msec): min=138, max=5937, avg=1714.07, stdev=979.92
    clat percentiles (msec):
     |  1.00th=[  776],  5.00th=[  953], 10.00th=[ 1020], 20.00th=[ 1116],
     | 30.00th=[ 1200], 40.00th=[ 1267], 50.00th=[ 1351], 60.00th=[ 1418],
     | 70.00th=[ 1536], 80.00th=[ 1770], 90.00th=[ 3507], 95.00th=[ 4044],
     | 99.00th=[ 4799], 99.50th=[ 5000], 99.90th=[ 5671], 99.95th=[ 5805],
     | 99.99th=[ 5873]
   bw (  MiB/s): min=  320, max= 5600, per=99.70%, avg=2365.68, stdev=304.50, samples=950
   iops        : min=   80, max= 1400, avg=591.17, stdev=76.12, samples=950
  lat (msec)   : 250=0.10%, 500=0.32%, 750=0.45%, 1000=7.49%, 2000=73.19%
  lat (msec)   : >=2000=18.45%
  cpu          : usr=0.04%, sys=1.49%, ctx=220456, majf=0, minf=4998
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.2%, >=64=99.6%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued rwts: total=71268,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=256

Run status group 0 (all jobs):
   READ: bw=2373MiB/s (2488MB/s), 2373MiB/s-2373MiB/s (2488MB/s-2488MB/s), io=278GiB (299GB), run=120138-120138msec

Disk stats (read/write):
  nvme1n1: ios=2272721/0, merge=0/0, ticks=585721705/0, in_queue=585721705, util=100.00%

Crucial P5 Plus 2TB PCIe M.2 2280SS not working in PCIe Gen4