Slow write speeds with Dell H200e

I’ve followed some guides but this has now gone beyond my limited knowledge.

I have a Dell R630 server running Proxmox V8.0.4 and I have installed a Dell H200e HAB flashed as per these instructions: -

https://techmattr.wordpress.com/2016/04/11/updated-sas-hba-crossflashing-or-flashing-to-it-mode-dell-perc-h200-and-h310/

And i’ve been trying to setup TrueNAS in a VM. Although I got it working the write performance was poor.

In an attempt to rule out issues with the hardware passthrough I have just done some read/write tests to a single drive in a MD1200 connected through the H200.

This is the read test:

root@proxmox:~# fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=4 --runtime=60 --time_based --name seq_read_job --filename=/dev/sdd
seq_read_job: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [R(1)][100.0%][r=168MiB/s][r=43.0k IOPS][eta 00m:00s]
seq_read_job: (groupid=0, jobs=1): err= 0: pid=10317: Sun Nov 12 14:12:12 2023
  read: IOPS=41.5k, BW=162MiB/s (170MB/s)(9734MiB/60001msec)
    slat (usec): min=3, max=675, avg=15.87, stdev= 5.57
    clat (usec): min=30, max=28442, avg=78.73, stdev=77.27
     lat (usec): min=41, max=28458, avg=94.60, stdev=77.14
    clat percentiles (usec):
     |  1.00th=[   43],  5.00th=[   54], 10.00th=[   63], 20.00th=[   70],
     | 30.00th=[   72], 40.00th=[   74], 50.00th=[   74], 60.00th=[   76],
     | 70.00th=[   79], 80.00th=[   81], 90.00th=[   85], 95.00th=[   89],
     | 99.00th=[  143], 99.50th=[  660], 99.90th=[  865], 99.95th=[  873],
     | 99.99th=[  955]
   bw (  KiB/s): min=124528, max=175392, per=100.00%, avg=166278.55, stdev=5530.75, samples=119
   iops        : min=31132, max=43848, avg=41569.62, stdev=1382.68, samples=119
  lat (usec)   : 50=3.37%, 100=95.08%, 250=0.71%, 500=0.32%, 750=0.03%
  lat (usec)   : 1000=0.48%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%
  cpu          : usr=9.57%, sys=40.60%, ctx=2888210, majf=0, minf=42
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=2492009,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
   READ: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=9734MiB (10.2GB), run=60001-60001msec

Disk stats (read/write):
  sdd: ios=2487331/0, merge=82/0, ticks=151526/0, in_queue=151526, util=99.95%

And the write test:

root@proxmox:~# fio --ioengine=libaio --direct=1 --sync=1 --rw=write --bs=4K --numjobs=1 --iodepth=4 --runtime=60 --time_based --name seq_write --filename=/dev/sdd 
seq_write: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4
fio-3.33
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=624KiB/s][w=156 IOPS][eta 00m:00s]
seq_write: (groupid=0, jobs=1): err= 0: pid=8502: Sun Nov 12 14:00:37 2023
  write: IOPS=152, BW=611KiB/s (626kB/s)(35.8MiB/60019msec); 0 zone resets
    slat (usec): min=6, max=25638, avg=20.06, stdev=267.78
    clat (usec): min=24487, max=50681, avg=26114.08, stdev=2881.03
     lat (usec): min=24527, max=58414, avg=26134.15, stdev=2899.79
    clat percentiles (usec):
     |  1.00th=[25035],  5.00th=[25035], 10.00th=[25035], 20.00th=[25035],
     | 30.00th=[25035], 40.00th=[25035], 50.00th=[25035], 60.00th=[25035],
     | 70.00th=[25035], 80.00th=[25035], 90.00th=[33424], 95.00th=[33424],
     | 99.00th=[33424], 99.50th=[33817], 99.90th=[41681], 99.95th=[50070],
     | 99.99th=[50594]
   bw (  KiB/s): min=  448, max=  640, per=99.96%, avg=611.76, stdev=23.26, samples=119
   iops        : min=  112, max=  160, avg=152.94, stdev= 5.82, samples=119
  lat (msec)   : 50=99.92%, 100=0.08%
  cpu          : usr=0.05%, sys=0.37%, ctx=2341, majf=0, minf=12
  IO depths    : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,9172,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=4

Run status group 0 (all jobs):
  WRITE: bw=611KiB/s (626kB/s), 611KiB/s-611KiB/s (626kB/s-626kB/s), io=35.8MiB (37.6MB), run=60019-60019msec

Disk stats (read/write):
  sdd: ios=77/9152, merge=0/0, ticks=958/238391, in_queue=239348, util=99.80%

As you can see my write speed is very slow though I don’t know if that is showing a problem with my hardware or an issue with the way I am testing it.

The HDD’s are refurbished Dell branded SAS drives, model HUS723030ALS640.

Any pointers on what could be wrong would be greatly appreciated.

Thanks.