# fdisk -l /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN Disk /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN: 447.1 GiB, 480103981056 bytes, 937703088 sectors Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disklabel type: gpt Disk identifier: 6D3A4DC6-00D0-A04E-8C57-CA430F774FCD Device Start End Sectors Size Type /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN-part1 2048 937684991 937682944 447.1G Solaris /usr & Apple ZFS /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN-part9 937684992 937701375 16384 8M Solaris reserved 1 # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --direct=1 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=/dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN-part1 temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=43.1MiB/s,w=0KiB/s][r=88.3k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=18915: Sun May 27 11:02:16 2018 read: IOPS=87.8k, BW=42.9MiB/s (44.9MB/s)(10.0GiB/238930msec) clat (usec): min=9, max=1153, avg=10.89, stdev= 1.57 lat (usec): min=9, max=1153, avg=10.93, stdev= 1.57 clat percentiles (nsec): | 1.00th=[10176], 5.00th=[10304], 10.00th=[10304], 20.00th=[10432], | 30.00th=[10560], 40.00th=[10688], 50.00th=[10688], 60.00th=[10816], | 70.00th=[10944], 80.00th=[11072], 90.00th=[11328], 95.00th=[11456], | 99.00th=[14656], 99.50th=[16768], 99.90th=[36096], 99.95th=[38656], | 99.99th=[40192] bw ( KiB/s): min=42765, max=44473, per=99.97%, avg=43872.98, stdev=264.09, samples=477 iops : min=85530, max=88946, avg=87746.04, stdev=528.17, samples=477 lat (usec) : 10=0.21%, 20=99.50%, 50=0.29%, 100=0.01%, 250=0.01% lat (usec) : 500=0.01%, 750=0.01% lat (msec) : 2=0.01% cpu : usr=3.07%, sys=14.03%, ctx=20971217, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=42.9MiB/s (44.9MB/s), 42.9MiB/s-42.9MiB/s (44.9MB/s-44.9MB/s), io=10.0GiB (10.7GB), run=238930-238930msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 7.53s user 33.88s system 17% cpu 3:59.35 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --direct=1 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=/dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN-part1 temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=294MiB/s,w=0KiB/s][r=75.2k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=18982: Sun May 27 11:05:50 2018 read: IOPS=74.8k, BW=292MiB/s (306MB/s)(10.0GiB/35038msec) clat (usec): min=11, max=448, avg=12.91, stdev= 1.54 lat (usec): min=11, max=448, avg=12.95, stdev= 1.54 clat percentiles (nsec): | 1.00th=[12224], 5.00th=[12352], 10.00th=[12480], 20.00th=[12480], | 30.00th=[12608], 40.00th=[12736], 50.00th=[12736], 60.00th=[12736], | 70.00th=[12864], 80.00th=[12864], 90.00th=[13120], 95.00th=[13888], | 99.00th=[16064], 99.50th=[18816], 99.90th=[38656], 99.95th=[40704], | 99.99th=[42240] bw ( KiB/s): min=283057, max=302080, per=99.93%, avg=299068.11, stdev=3591.27, samples=70 iops : min=70764, max=75520, avg=74766.93, stdev=897.92, samples=70 lat (usec) : 20=99.60%, 50=0.40%, 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=2.27%, sys=11.79%, ctx=2621420, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=292MiB/s (306MB/s), 292MiB/s-292MiB/s (306MB/s-306MB/s), io=10.0GiB (10.7GB), run=35038-35038msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k 0.88s user 4.39s system 14% cpu 35.454 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --direct=1 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=/dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN-part1 temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=1197MiB/s,w=0KiB/s][r=19.2k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=19038: Sun May 27 11:09:52 2018 read: IOPS=19.1k, BW=1195MiB/s (1253MB/s)(10.0GiB/8568msec) clat (usec): min=50, max=776, avg=51.83, stdev= 3.17 lat (usec): min=50, max=776, avg=51.87, stdev= 3.17 clat percentiles (nsec): | 1.00th=[50944], 5.00th=[50944], 10.00th=[50944], 20.00th=[51456], | 30.00th=[51456], 40.00th=[51456], 50.00th=[51456], 60.00th=[51456], | 70.00th=[51456], 80.00th=[51968], 90.00th=[51968], 95.00th=[52992], | 99.00th=[63232], 99.50th=[72192], 99.90th=[79360], 99.95th=[80384], | 99.99th=[83456] bw ( MiB/s): min= 1192, max= 1198, per=99.99%, avg=1194.98, stdev= 1.80, samples=17 iops : min=19076, max=19170, avg=19119.65, stdev=28.92, samples=17 lat (usec) : 100=100.00%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=0.06%, sys=5.00%, ctx=163842, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1195MiB/s (1253MB/s), 1195MiB/s-1195MiB/s (1253MB/s-1253MB/s), io=10.0GiB (10.7GB), run=8568-8568msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --direct=1 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=/dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN-part1 temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process Jobs: 1 (f=1): [r(1)][100.0%][r=1578MiB/s,w=0KiB/s][r=1577,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=19082: Sun May 27 11:13:30 2018 read: IOPS=1577, BW=1578MiB/s (1654MB/s)(10.0GiB/6491msec) clat (usec): min=629, max=876, avg=633.22, stdev= 5.59 lat (usec): min=630, max=877, avg=633.26, stdev= 5.60 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 627], 10.00th=[ 627], 20.00th=[ 627], | 30.00th=[ 627], 40.00th=[ 635], 50.00th=[ 635], 60.00th=[ 635], | 70.00th=[ 635], 80.00th=[ 635], 90.00th=[ 644], 95.00th=[ 644], | 99.00th=[ 652], 99.50th=[ 652], 99.90th=[ 660], 99.95th=[ 668], | 99.99th=[ 668] bw ( MiB/s): min= 1576, max= 1580, per=100.00%, avg=1578.00, stdev= 1.91, samples=12 iops : min= 1576, max= 1580, avg=1578.00, stdev= 1.91, samples=12 lat (usec) : 750=99.99%, 1000=0.01% cpu : usr=0.03%, sys=3.84%, ctx=10240, majf=0, minf=265 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1578MiB/s (1654MB/s), 1578MiB/s-1578MiB/s (1654MB/s-1654MB/s), io=10.0GiB (10.7GB), run=6491-6491msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --direct=1 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=/dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN-part1 temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process Jobs: 1 (f=1): [R(1)][100.0%][r=1581MiB/s,w=0KiB/s][r=1580,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=19157: Sun May 27 11:17:18 2018 read: IOPS=1578, BW=1579MiB/s (1655MB/s)(10.0GiB/6487msec) clat (usec): min=629, max=1066, avg=632.99, stdev= 6.95 lat (usec): min=629, max=1066, avg=633.04, stdev= 6.95 clat percentiles (usec): | 1.00th=[ 627], 5.00th=[ 627], 10.00th=[ 627], 20.00th=[ 627], | 30.00th=[ 627], 40.00th=[ 627], 50.00th=[ 635], 60.00th=[ 635], | 70.00th=[ 635], 80.00th=[ 635], 90.00th=[ 644], 95.00th=[ 644], | 99.00th=[ 652], 99.50th=[ 652], 99.90th=[ 660], 99.95th=[ 660], | 99.99th=[ 865] bw ( MiB/s): min= 1574, max= 1580, per=100.00%, avg=1578.57, stdev= 1.54, samples=12 iops : min= 1574, max= 1580, avg=1578.50, stdev= 1.73, samples=12 lat (usec) : 750=99.97%, 1000=0.02% lat (msec) : 2=0.01% cpu : usr=0.19%, sys=3.44%, ctx=10255, majf=0, minf=265 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1579MiB/s (1655MB/s), 1579MiB/s-1579MiB/s (1655MB/s-1655MB/s), io=10.0GiB (10.7GB), run=6487-6487msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --direct=1 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=/dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN-part1 temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=1438MiB/s][r=0,w=1438 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=19546: Sun May 27 12:16:59 2018 write: IOPS=1436, BW=1436MiB/s (1506MB/s)(10.0GiB/7129msec) clat (usec): min=658, max=809, avg=692.27, stdev=24.11 lat (usec): min=661, max=812, avg=695.64, stdev=24.18 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 660], 10.00th=[ 660], 20.00th=[ 676], | 30.00th=[ 676], 40.00th=[ 676], 50.00th=[ 693], 60.00th=[ 701], | 70.00th=[ 701], 80.00th=[ 717], 90.00th=[ 725], 95.00th=[ 734], | 99.00th=[ 758], 99.50th=[ 766], 99.90th=[ 791], 99.95th=[ 799], | 99.99th=[ 799] bw ( MiB/s): min= 1430, max= 1440, per=100.00%, avg=1436.37, stdev= 3.63, samples=14 iops : min= 1430, max= 1440, avg=1436.36, stdev= 3.63, samples=14 lat (usec) : 750=97.83%, 1000=2.17% cpu : usr=0.56%, sys=3.10%, ctx=10260, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=1436MiB/s (1506MB/s), 1436MiB/s-1436MiB/s (1506MB/s-1506MB/s), io=10.0GiB (10.7GB), run=7129-7129msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --direct=1 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=/dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN-part1 temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=1436MiB/s][r=0,w=1436 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=19579: Sun May 27 12:18:25 2018 write: IOPS=1436, BW=1436MiB/s (1506MB/s)(10.0GiB/7130msec) clat (usec): min=658, max=1283, avg=692.17, stdev=24.62 lat (usec): min=661, max=1287, avg=695.50, stdev=24.63 clat percentiles (usec): | 1.00th=[ 660], 5.00th=[ 660], 10.00th=[ 660], 20.00th=[ 676], | 30.00th=[ 676], 40.00th=[ 685], 50.00th=[ 693], 60.00th=[ 701], | 70.00th=[ 701], 80.00th=[ 709], 90.00th=[ 725], 95.00th=[ 734], | 99.00th=[ 758], 99.50th=[ 766], 99.90th=[ 791], 99.95th=[ 799], | 99.99th=[ 824] bw ( MiB/s): min= 1432, max= 1440, per=100.00%, avg=1436.22, stdev= 2.41, samples=14 iops : min= 1432, max= 1440, avg=1436.21, stdev= 2.42, samples=14 lat (usec) : 750=97.98%, 1000=2.01% lat (msec) : 2=0.01% cpu : usr=1.00%, sys=2.58%, ctx=10242, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=1436MiB/s (1506MB/s), 1436MiB/s-1436MiB/s (1506MB/s-1506MB/s), io=10.0GiB (10.7GB), run=7130-7130msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --direct=1 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=/dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN-part1 temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=1036MiB/s][r=0,w=16.6k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=19608: Sun May 27 12:19:19 2018 write: IOPS=16.4k, BW=1025MiB/s (1075MB/s)(10.0GiB/9992msec) clat (usec): min=55, max=763, avg=60.20, stdev= 8.79 lat (usec): min=55, max=763, avg=60.47, stdev= 8.81 clat percentiles (usec): | 1.00th=[ 56], 5.00th=[ 57], 10.00th=[ 57], 20.00th=[ 57], | 30.00th=[ 57], 40.00th=[ 57], 50.00th=[ 58], 60.00th=[ 58], | 70.00th=[ 58], 80.00th=[ 62], 90.00th=[ 66], 95.00th=[ 85], | 99.00th=[ 92], 99.50th=[ 95], 99.90th=[ 108], 99.95th=[ 113], | 99.99th=[ 119] bw ( MiB/s): min= 940, max= 1043, per=99.94%, avg=1024.23, stdev=30.93, samples=19 iops : min=15054, max=16700, avg=16387.68, stdev=494.90, samples=19 lat (usec) : 100=99.83%, 250=0.17%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=0.70%, sys=3.55%, ctx=163867, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=1025MiB/s (1075MB/s), 1025MiB/s-1025MiB/s (1075MB/s-1075MB/s), io=10.0GiB (10.7GB), run=9992-9992msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --direct=1 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=/dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN-part1 temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=233MiB/s][r=0,w=59.7k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=19636: Sun May 27 12:21:14 2018 write: IOPS=59.5k, BW=232MiB/s (244MB/s)(10.0GiB/44045msec) clat (usec): min=15, max=913, avg=16.26, stdev= 2.65 lat (usec): min=15, max=913, avg=16.31, stdev= 2.66 clat percentiles (nsec): | 1.00th=[15424], 5.00th=[15552], 10.00th=[15552], 20.00th=[15808], | 30.00th=[15808], 40.00th=[15936], 50.00th=[15936], 60.00th=[15936], | 70.00th=[16064], 80.00th=[16064], 90.00th=[16512], 95.00th=[17024], | 99.00th=[31104], 99.50th=[41728], 99.90th=[43776], 99.95th=[45312], | 99.99th=[46848] bw ( KiB/s): min=228878, max=239544, per=99.96%, avg=237974.91, stdev=2025.14, samples=88 iops : min=57219, max=59886, avg=59493.64, stdev=506.31, samples=88 lat (usec) : 20=98.48%, 50=1.52%, 100=0.01%, 500=0.01%, 750=0.01% lat (usec) : 1000=0.01% cpu : usr=1.41%, sys=10.96%, ctx=2621760, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=232MiB/s (244MB/s), 232MiB/s-232MiB/s (244MB/s-244MB/s), io=10.0GiB (10.7GB), run=44045-44045msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k 0.71s user 5.10s system 13% cpu 44.453 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --direct=1 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=/dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN-part1 temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=24.6MiB/s][r=0,w=50.4k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=19674: Sun May 27 12:29:29 2018 write: IOPS=55.6k, BW=27.2MiB/s (28.5MB/s)(10.0GiB/376988msec) clat (usec): min=15, max=1167, avg=17.46, stdev= 3.80 lat (usec): min=15, max=1167, avg=17.51, stdev= 3.80 clat percentiles (nsec): | 1.00th=[16320], 5.00th=[16512], 10.00th=[16512], 20.00th=[16768], | 30.00th=[16768], 40.00th=[16768], 50.00th=[17024], 60.00th=[17024], | 70.00th=[17024], 80.00th=[17024], 90.00th=[17536], 95.00th=[18304], | 99.00th=[46848], 99.50th=[48384], 99.90th=[52480], 99.95th=[52480], | 99.99th=[54016] bw ( KiB/s): min=25460, max=28045, per=99.99%, avg=27810.22, stdev=211.95, samples=753 iops : min=50920, max=56090, avg=55620.47, stdev=423.89, samples=753 lat (usec) : 20=97.83%, 50=1.71%, 100=0.46%, 250=0.01%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=1.23%, sys=10.49%, ctx=20974606, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=27.2MiB/s (28.5MB/s), 27.2MiB/s-27.2MiB/s (28.5MB/s-28.5MB/s), io=10.0GiB (10.7GB), run=376988-376988msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=51 4.86s user 40.03s system 11% cpu 6:17.42 total zpool create -o ashift=9 myzpool /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN zfs create myzpool/mydataset zfs set recordsize=128K myzpool/mydataset # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][99.8%][r=54.7MiB/s,w=0KiB/s][r=112k,w=0 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=17181: Sun May 27 13:14:12 2018 read: IOPS=46.0k, BW=22.9MiB/s (24.1MB/s)(10.0GiB/446320msec) clat (nsec): min=1470, max=1103.2k, avg=20701.03, stdev=5074.74 lat (nsec): min=1510, max=1103.2k, avg=20739.92, stdev=5075.78 clat percentiles (nsec): | 1.00th=[ 2480], 5.00th=[18816], 10.00th=[19328], 20.00th=[19840], | 30.00th=[20096], 40.00th=[20608], 50.00th=[20864], 60.00th=[21120], | 70.00th=[21632], 80.00th=[22144], 90.00th=[22656], 95.00th=[23424], | 99.00th=[25472], 99.50th=[40192], 99.90th=[44288], 99.95th=[45824], | 99.99th=[61696] bw ( KiB/s): min=21299, max=64820, per=99.69%, avg=23420.70, stdev=2027.30, samples=892 iops : min=42598, max=129640, avg=46841.49, stdev=4054.60, samples=892 lat (usec) : 2=0.55%, 4=1.88%, 10=0.01%, 20=21.87%, 50=75.69% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=3.10%, sys=96.50%, ctx=1869, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=22.9MiB/s (24.1MB/s), 22.9MiB/s-22.9MiB/s (24.1MB/s-24.1MB/s), io=10.0GiB (10.7GB), run=446320-446320msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 16.61s user 501.59s system 99% cpu 8:39.90 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][98.3%][r=247MiB/s,w=0KiB/s][r=63.3k,w=0 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=17885: Sun May 27 13:16:23 2018 read: IOPS=45.4k, BW=177MiB/s (186MB/s)(10.0GiB/57734msec) clat (nsec): min=1500, max=1759.8k, avg=21409.72, stdev=5390.37 lat (nsec): min=1540, max=1759.9k, avg=21450.44, stdev=5391.34 clat percentiles (nsec): | 1.00th=[ 2288], 5.00th=[ 3760], 10.00th=[20352], 20.00th=[21120], | 30.00th=[21632], 40.00th=[21888], 50.00th=[22144], 60.00th=[22400], | 70.00th=[22656], 80.00th=[23168], 90.00th=[23936], 95.00th=[24448], | 99.00th=[26496], 99.50th=[42240], 99.90th=[46336], 99.95th=[47360], | 99.99th=[49920] bw ( KiB/s): min=171616, max=343489, per=99.01%, avg=179823.70, stdev=19942.46, samples=115 iops : min=42904, max=85872, avg=44955.84, stdev=4985.62, samples=115 lat (usec) : 2=0.64%, 4=4.51%, 10=0.07%, 20=2.83%, 50=91.94% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=2.97%, sys=96.69%, ctx=124, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=177MiB/s (186MB/s), 177MiB/s-177MiB/s (186MB/s-186MB/s), io=10.0GiB (10.7GB), run=57734-57734msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k 3.35s user 68.34s system 99% cpu 1:12.25 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][80.0%][r=2571MiB/s,w=0KiB/s][r=41.1k,w=0 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=11539: Sun May 27 13:18:05 2018 read: IOPS=41.3k, BW=2581MiB/s (2707MB/s)(10.0GiB/3967msec) clat (usec): min=3, max=168, avg=23.58, stdev= 3.97 lat (usec): min=3, max=168, avg=23.62, stdev= 3.97 clat percentiles (nsec): | 1.00th=[ 5408], 5.00th=[21632], 10.00th=[22400], 20.00th=[22912], | 30.00th=[23424], 40.00th=[23680], 50.00th=[23936], 60.00th=[24192], | 70.00th=[24448], 80.00th=[24960], 90.00th=[25728], 95.00th=[26240], | 99.00th=[28544], 99.50th=[42240], 99.90th=[45312], 99.95th=[47360], | 99.99th=[79360] bw ( MiB/s): min= 2495, max= 2616, per=98.91%, avg=2553.04, stdev=37.88, samples=7 iops : min=39934, max=41860, avg=40848.57, stdev=606.00, samples=7 lat (usec) : 4=0.04%, 10=3.66%, 20=0.18%, 50=96.08%, 100=0.04% lat (usec) : 250=0.01% cpu : usr=3.43%, sys=96.29%, ctx=6, majf=0, minf=23 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=2581MiB/s (2707MB/s), 2581MiB/s-2581MiB/s (2707MB/s-2707MB/s), io=10.0GiB (10.7GB), run=3967-3967msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k 1.72s user 8.37s system 68% cpu 14.710 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1) temp: (groupid=0, jobs=1): err= 0: pid=1301: Sun May 27 13:19:41 2018 read: IOPS=7125, BW=7126MiB/s (7472MB/s)(10.0GiB/1437msec) clat (usec): min=126, max=747, avg=139.68, stdev=11.80 lat (usec): min=126, max=747, avg=139.72, stdev=11.80 clat percentiles (usec): | 1.00th=[ 130], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 135], | 30.00th=[ 137], 40.00th=[ 137], 50.00th=[ 137], 60.00th=[ 139], | 70.00th=[ 141], 80.00th=[ 143], 90.00th=[ 149], 95.00th=[ 155], | 99.00th=[ 174], 99.50th=[ 192], 99.90th=[ 253], 99.95th=[ 260], | 99.99th=[ 273] bw ( MiB/s): min= 7064, max= 7178, per=99.93%, avg=7121.00, stdev=80.61, samples=2 iops : min= 7064, max= 7178, avg=7121.00, stdev=80.61, samples=2 lat (usec) : 250=99.88%, 500=0.11%, 750=0.01% cpu : usr=0.21%, sys=99.58%, ctx=6, majf=0, minf=266 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=7126MiB/s (7472MB/s), 7126MiB/s-7126MiB/s (7472MB/s-7472MB/s), io=10.0GiB (10.7GB), run=1437-1437msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m 2.08s user 4.79s system 61% cpu 11.115 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp :( temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process Jobs: 1 (f=1) temp: (groupid=0, jobs=1): err= 0: pid=23404: Sun May 27 13:29:25 2018 read: IOPS=7180, BW=7181MiB/s (7530MB/s)(10.0GiB/1426msec) clat (usec): min=123, max=526, avg=138.84, stdev= 9.10 lat (usec): min=123, max=526, avg=138.88, stdev= 9.11 clat percentiles (usec): | 1.00th=[ 131], 5.00th=[ 133], 10.00th=[ 135], 20.00th=[ 135], | 30.00th=[ 137], 40.00th=[ 137], 50.00th=[ 139], 60.00th=[ 139], | 70.00th=[ 141], 80.00th=[ 141], 90.00th=[ 143], 95.00th=[ 145], | 99.00th=[ 176], 99.50th=[ 184], 99.90th=[ 255], 99.95th=[ 258], | 99.99th=[ 269] bw ( MiB/s): min= 7114, max= 7224, per=99.83%, avg=7169.00, stdev=77.78, samples=2 iops : min= 7114, max= 7224, avg=7169.00, stdev=77.78, samples=2 lat (usec) : 250=99.84%, 500=0.15%, 750=0.01% cpu : usr=0.21%, sys=99.58%, ctx=3, majf=0, minf=267 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=7181MiB/s (7530MB/s), 7181MiB/s-7181MiB/s (7530MB/s-7530MB/s), io=10.0GiB (10.7GB), run=1426-1426msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=872MiB/s][r=0,w=872 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=23484: Sun May 27 13:30:48 2018 write: IOPS=877, BW=877MiB/s (920MB/s)(10.0GiB/11670msec) clat (usec): min=923, max=4524, avg=1112.90, stdev=142.54 lat (usec): min=934, max=4550, avg=1137.85, stdev=143.33 clat percentiles (usec): | 1.00th=[ 963], 5.00th=[ 988], 10.00th=[ 1012], 20.00th=[ 1037], | 30.00th=[ 1057], 40.00th=[ 1074], 50.00th=[ 1090], 60.00th=[ 1106], | 70.00th=[ 1123], 80.00th=[ 1156], 90.00th=[ 1188], 95.00th=[ 1270], | 99.00th=[ 1876], 99.50th=[ 2008], 99.90th=[ 2114], 99.95th=[ 2180], | 99.99th=[ 2245] bw ( KiB/s): min=886784, max=921600, per=99.96%, avg=898181.57, stdev=9258.85, samples=23 iops : min= 866, max= 900, avg=877.13, stdev= 9.04, samples=23 lat (usec) : 1000=7.36% lat (msec) : 2=92.12%, 4=0.51%, 10=0.01% cpu : usr=2.79%, sys=26.99%, ctx=21659, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=877MiB/s (920MB/s), 877MiB/s-877MiB/s (920MB/s-920MB/s), io=10.0GiB (10.7GB), run=11670-11670msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=863MiB/s][r=0,w=862 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=27420: Sun May 27 13:33:06 2018 write: IOPS=826, BW=826MiB/s (866MB/s)(10.0GiB/12392msec) clat (usec): min=923, max=2387, avg=1181.60, stdev=156.74 lat (usec): min=949, max=2415, avg=1207.41, stdev=158.53 clat percentiles (usec): | 1.00th=[ 979], 5.00th=[ 1020], 10.00th=[ 1045], 20.00th=[ 1074], | 30.00th=[ 1106], 40.00th=[ 1123], 50.00th=[ 1156], 60.00th=[ 1172], | 70.00th=[ 1205], 80.00th=[ 1254], 90.00th=[ 1319], 95.00th=[ 1418], | 99.00th=[ 1909], 99.50th=[ 2057], 99.90th=[ 2245], 99.95th=[ 2278], | 99.99th=[ 2311] bw ( KiB/s): min=804864, max=892928, per=99.84%, avg=844829.29, stdev=29477.28, samples=24 iops : min= 786, max= 872, avg=825.00, stdev=28.79, samples=24 lat (usec) : 1000=2.71% lat (msec) : 2=96.60%, 4=0.69% cpu : usr=2.34%, sys=26.91%, ctx=21319, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=826MiB/s (866MB/s), 826MiB/s-826MiB/s (866MB/s-866MB/s), io=10.0GiB (10.7GB), run=12392-12392msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=230MiB/s][r=0,w=3677 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=6141: Sun May 27 13:35:02 2018 write: IOPS=3672, BW=230MiB/s (241MB/s)(10.0GiB/44614msec) clat (usec): min=168, max=3345, avg=268.88, stdev=108.21 lat (usec): min=169, max=3348, avg=270.22, stdev=108.29 clat percentiles (usec): | 1.00th=[ 198], 5.00th=[ 208], 10.00th=[ 219], 20.00th=[ 229], | 30.00th=[ 239], 40.00th=[ 247], 50.00th=[ 253], 60.00th=[ 262], | 70.00th=[ 269], 80.00th=[ 281], 90.00th=[ 297], 95.00th=[ 326], | 99.00th=[ 1037], 99.50th=[ 1156], 99.90th=[ 1270], 99.95th=[ 1336], | 99.99th=[ 2040] bw ( KiB/s): min=215808, max=253696, per=99.99%, avg=235007.82, stdev=8898.70, samples=89 iops : min= 3372, max= 3964, avg=3671.99, stdev=139.04, samples=89 lat (usec) : 250=44.40%, 500=53.92%, 750=0.41%, 1000=0.17% lat (msec) : 2=1.08%, 4=0.01% cpu : usr=2.06%, sys=23.85%, ctx=332902, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=230MiB/s (241MB/s), 230MiB/s-230MiB/s (241MB/s-241MB/s), io=10.0GiB (10.7GB), run=44614-44614msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64 0.98s user 10.95s system 26% cpu 45.020 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][99.8%][r=0KiB/s,w=35.5MiB/s][r=0,w=9080 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=7636: Sun May 27 13:44:23 2018 write: IOPS=5654, BW=22.1MiB/s (23.2MB/s)(10.0GiB/463611msec) clat (usec): min=47, max=100232, avg=174.63, stdev=152.91 lat (usec): min=47, max=100232, avg=174.90, stdev=152.96 clat percentiles (usec): | 1.00th=[ 73], 5.00th=[ 84], 10.00th=[ 88], 20.00th=[ 97], | 30.00th=[ 108], 40.00th=[ 116], 50.00th=[ 129], 60.00th=[ 190], | 70.00th=[ 221], 80.00th=[ 255], 90.00th=[ 285], 95.00th=[ 306], | 99.00th=[ 375], 99.50th=[ 578], 99.90th=[ 1991], 99.95th=[ 2573], | 99.99th=[ 4146] bw ( KiB/s): min=18000, max=40608, per=99.98%, avg=22613.40, stdev=1566.18, samples=927 iops : min= 4500, max=10152, avg=5653.33, stdev=391.54, samples=927 lat (usec) : 50=0.01%, 100=23.50%, 250=54.93%, 500=21.00%, 750=0.18% lat (usec) : 1000=0.17% lat (msec) : 2=0.13%, 4=0.09%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 250=0.01% cpu : usr=2.16%, sys=32.02%, ctx=5294371, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=22.1MiB/s (23.2MB/s), 22.1MiB/s-22.1MiB/s (23.2MB/s-23.2MB/s), io=10.0GiB (10.7GB), run=463611-463611msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k 10.56s user 149.42s system 34% cpu 7:44.06 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=6076KiB/s][r=0,w=12.2k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=23816: Sun May 27 15:14:38 2018 write: IOPS=5741, BW=2871KiB/s (2940kB/s)(10.0GiB/3652717msec) clat (usec): min=41, max=106370, avg=171.92, stdev=157.10 lat (usec): min=41, max=106371, avg=172.12, stdev=157.13 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 82], 10.00th=[ 86], 20.00th=[ 95], | 30.00th=[ 105], 40.00th=[ 113], 50.00th=[ 133], 60.00th=[ 192], | 70.00th=[ 221], 80.00th=[ 245], 90.00th=[ 273], 95.00th=[ 293], | 99.00th=[ 359], 99.50th=[ 570], 99.90th=[ 2024], 99.95th=[ 2606], | 99.99th=[ 4146] bw ( KiB/s): min= 2204, max= 6110, per=100.00%, avg=2870.12, stdev=156.19, samples=7305 iops : min= 4408, max=12220, avg=5740.26, stdev=312.38, samples=7305 lat (usec) : 50=0.01%, 100=24.83%, 250=57.43%, 500=17.18%, 750=0.16% lat (usec) : 1000=0.17% lat (msec) : 2=0.12%, 4=0.09%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=2.16%, sys=31.71%, ctx=42331006, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=2871KiB/s (2940kB/s), 2871KiB/s-2871KiB/s (2940kB/s-2940kB/s), io=10.0GiB (10.7GB), run=3652717-3652717msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=51 82.43s user 1164.36s system 34% cpu 1:00:53.46 total zpool create -o ashift=9 myzpool /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN zfs create myzpool/mydataset zfs set recordsize=4K myzpool/mydataset # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=144MiB/s,w=0KiB/s][r=295k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=6223: Sun May 27 20:49:44 2018 read: IOPS=199k, BW=97.4MiB/s (102MB/s)(10.0GiB/105133msec) clat (nsec): min=1610, max=7136.3k, avg=4532.94, stdev=2367.04 lat (nsec): min=1649, max=7136.4k, avg=4569.83, stdev=2367.28 clat percentiles (nsec): | 1.00th=[ 2024], 5.00th=[ 2608], 10.00th=[ 3568], 20.00th=[ 3888], | 30.00th=[ 4080], 40.00th=[ 4192], 50.00th=[ 4384], 60.00th=[ 4512], | 70.00th=[ 4896], 80.00th=[ 5472], 90.00th=[ 5920], 95.00th=[ 6176], | 99.00th=[ 6688], 99.50th=[ 6880], 99.90th=[ 9920], 99.95th=[11712], | 99.99th=[34048] bw ( KiB/s): min=74349, max=153336, per=99.84%, avg=99574.80, stdev=7563.69, samples=210 iops : min=148698, max=306672, avg=199149.93, stdev=15127.38, samples=210 lat (usec) : 2=0.97%, 4=24.76%, 10=74.17%, 20=0.08%, 50=0.01% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=10.22%, sys=89.66%, ctx=633, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=97.4MiB/s (102MB/s), 97.4MiB/s-97.4MiB/s (102MB/s-102MB/s), io=10.0GiB (10.7GB), run=105133-105133msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 13.56s user 177.15s system 94% cpu 3:20.92 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=765MiB/s,w=0KiB/s][r=196k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=26384: Sun May 27 20:44:34 2018 read: IOPS=186k, BW=727MiB/s (762MB/s)(10.0GiB/14094msec) clat (usec): min=2, max=2821, avg= 4.92, stdev= 3.66 lat (usec): min=2, max=2821, avg= 4.96, stdev= 3.66 clat percentiles (nsec): | 1.00th=[ 3568], 5.00th=[ 3792], 10.00th=[ 3952], 20.00th=[ 4192], | 30.00th=[ 4320], 40.00th=[ 4512], 50.00th=[ 4640], 60.00th=[ 4832], | 70.00th=[ 5216], 80.00th=[ 5728], 90.00th=[ 6112], 95.00th=[ 6368], | 99.00th=[ 6880], 99.50th=[ 7200], 99.90th=[33024], 99.95th=[60672], | 99.99th=[68096] bw ( KiB/s): min=579057, max=811376, per=99.82%, avg=742630.93, stdev=36340.76, samples=28 iops : min=144764, max=202844, avg=185657.75, stdev=9085.24, samples=28 lat (usec) : 4=11.39%, 10=88.41%, 20=0.10%, 50=0.05%, 100=0.05% lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=10.10%, sys=89.58%, ctx=539, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=727MiB/s (762MB/s), 727MiB/s-727MiB/s (762MB/s-762MB/s), io=10.0GiB (10.7GB), run=14094-14094msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k 3.18s user 30.41s system 77% cpu 43.081 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=1112MiB/s,w=0KiB/s][r=17.8k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=17485: Sun May 27 20:42:40 2018 read: IOPS=16.9k, BW=1055MiB/s (1106MB/s)(10.0GiB/9710msec) clat (usec): min=13, max=9291, avg=58.68, stdev=37.97 lat (usec): min=13, max=9291, avg=58.73, stdev=37.97 clat percentiles (usec): | 1.00th=[ 36], 5.00th=[ 37], 10.00th=[ 38], 20.00th=[ 39], | 30.00th=[ 48], 40.00th=[ 52], 50.00th=[ 55], 60.00th=[ 67], | 70.00th=[ 69], 80.00th=[ 80], 90.00th=[ 84], 95.00th=[ 85], | 99.00th=[ 90], 99.50th=[ 95], 99.90th=[ 129], 99.95th=[ 141], | 99.99th=[ 562] bw ( MiB/s): min= 930, max= 1215, per=99.42%, avg=1048.43, stdev=57.62, samples=19 iops : min=14884, max=19440, avg=16774.84, stdev=921.96, samples=19 lat (usec) : 20=0.14%, 50=32.29%, 100=67.24%, 250=0.32%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=1.16%, sys=98.20%, ctx=281, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1055MiB/s (1106MB/s), 1055MiB/s-1055MiB/s (1106MB/s-1106MB/s), io=10.0GiB (10.7GB), run=9710-9710msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k 1.67s user 26.34s system 74% cpu 37.741 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=1478MiB/s,w=0KiB/s][r=1478,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=4741: Sun May 27 20:40:13 2018 read: IOPS=1464, BW=1464MiB/s (1535MB/s)(10.0GiB/6993msec) clat (usec): min=186, max=2682, avg=682.11, stdev=128.67 lat (usec): min=186, max=2682, avg=682.16, stdev=128.67 clat percentiles (usec): | 1.00th=[ 510], 5.00th=[ 529], 10.00th=[ 545], 20.00th=[ 553], | 30.00th=[ 570], 40.00th=[ 586], 50.00th=[ 734], 60.00th=[ 775], | 70.00th=[ 791], 80.00th=[ 799], 90.00th=[ 807], 95.00th=[ 816], | 99.00th=[ 857], 99.50th=[ 988], 99.90th=[ 1450], 99.95th=[ 1713], | 99.99th=[ 2507] bw ( MiB/s): min= 1390, max= 1498, per=99.84%, avg=1462.01, stdev=29.62, samples=13 iops : min= 1390, max= 1498, avg=1462.00, stdev=29.62, samples=13 lat (usec) : 250=0.21%, 500=0.27%, 750=51.79%, 1000=47.26% lat (msec) : 2=0.43%, 4=0.04% cpu : usr=0.19%, sys=99.59%, ctx=139, majf=0, minf=265 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1464MiB/s (1535MB/s), 1464MiB/s-1464MiB/s (1535MB/s-1535MB/s), io=10.0GiB (10.7GB), run=6993-6993msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m 1.63s user 23.72s system 71% cpu 35.434 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [R(1)][100.0%][r=1100MiB/s,w=0KiB/s][r=1100,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=22901: Sun May 27 20:37:48 2018 read: IOPS=1069, BW=1069MiB/s (1121MB/s)(10.0GiB/9575msec) clat (usec): min=557, max=6869, avg=934.53, stdev=173.75 lat (usec): min=557, max=6869, avg=934.58, stdev=173.75 clat percentiles (usec): | 1.00th=[ 750], 5.00th=[ 766], 10.00th=[ 775], 20.00th=[ 783], | 30.00th=[ 799], 40.00th=[ 816], 50.00th=[ 865], 60.00th=[ 1037], | 70.00th=[ 1057], 80.00th=[ 1090], 90.00th=[ 1106], 95.00th=[ 1139], | 99.00th=[ 1237], 99.50th=[ 1270], 99.90th=[ 1663], 99.95th=[ 2180], | 99.99th=[ 6128] bw ( MiB/s): min= 948, max= 1102, per=99.93%, avg=1068.70, stdev=36.27, samples=19 iops : min= 948, max= 1102, avg=1068.53, stdev=36.20, samples=19 lat (usec) : 750=0.76%, 1000=53.99% lat (msec) : 2=45.17%, 4=0.05%, 10=0.03% cpu : usr=0.14%, sys=99.58%, ctx=47, majf=0, minf=264 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1069MiB/s (1121MB/s), 1069MiB/s-1069MiB/s (1121MB/s-1121MB/s), io=10.0GiB (10.7GB), run=9575-9575msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m 1.54s user 26.44s system 73% cpu 37.983 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=252MiB/s][r=0,w=252 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=28470: Sun May 27 20:34:25 2018 write: IOPS=250, BW=251MiB/s (263MB/s)(10.0GiB/40813msec) clat (usec): min=2276, max=72767, avg=3959.21, stdev=5175.93 lat (usec): min=2296, max=72791, avg=3983.66, stdev=5179.27 clat percentiles (usec): | 1.00th=[ 2343], 5.00th=[ 2409], 10.00th=[ 2442], 20.00th=[ 2540], | 30.00th=[ 2606], 40.00th=[ 2671], 50.00th=[ 2704], 60.00th=[ 2835], | 70.00th=[ 3097], 80.00th=[ 3359], 90.00th=[ 3884], 95.00th=[10683], | 99.00th=[33817], 99.50th=[40109], 99.90th=[54789], 99.95th=[62129], | 99.99th=[69731] bw ( KiB/s): min=235520, max=280576, per=99.90%, avg=256676.28, stdev=9970.82, samples=81 iops : min= 230, max= 274, avg=250.62, stdev= 9.74, samples=81 lat (msec) : 4=90.46%, 10=4.23%, 20=3.31%, 50=1.82%, 100=0.19% cpu : usr=0.64%, sys=57.11%, ctx=88110, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=251MiB/s (263MB/s), 251MiB/s-251MiB/s (263MB/s-263MB/s), io=10.0GiB (10.7GB), run=40813-40813msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m 0.33s user 23.64s system 58% cpu 41.227 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=242MiB/s][r=0,w=242 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=10755: Sun May 27 20:31:13 2018 write: IOPS=239, BW=239MiB/s (251MB/s)(10.0GiB/42764msec) clat (usec): min=2385, max=86235, avg=4149.04, stdev=5177.16 lat (usec): min=2402, max=86274, avg=4173.28, stdev=5181.78 clat percentiles (usec): | 1.00th=[ 2507], 5.00th=[ 2573], 10.00th=[ 2638], 20.00th=[ 2737], | 30.00th=[ 2802], 40.00th=[ 2835], 50.00th=[ 2900], 60.00th=[ 2966], | 70.00th=[ 3228], 80.00th=[ 3556], 90.00th=[ 4113], 95.00th=[11207], | 99.00th=[33424], 99.50th=[41681], 99.90th=[51643], 99.95th=[55313], | 99.99th=[65274] bw ( KiB/s): min=219136, max=294912, per=99.97%, avg=245123.01, stdev=12104.76, samples=85 iops : min= 214, max= 288, avg=239.35, stdev=11.82, samples=85 lat (msec) : 4=89.21%, 10=5.30%, 20=3.33%, 50=2.01%, 100=0.15% cpu : usr=0.66%, sys=58.89%, ctx=77807, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=239MiB/s (251MB/s), 239MiB/s-239MiB/s (251MB/s-251MB/s), io=10.0GiB (10.7GB), run=42764-42764msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m 0.34s user 25.51s system 59% cpu 43.174 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][98.4%][r=0KiB/s,w=163MiB/s][r=0,w=2605 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=6206: Sun May 27 20:28:57 2018 write: IOPS=2606, BW=163MiB/s (171MB/s)(10.0GiB/62853msec) clat (usec): min=193, max=35875, avg=380.54, stdev=975.55 lat (usec): min=194, max=35877, avg=381.96, stdev=975.73 clat percentiles (usec): | 1.00th=[ 212], 5.00th=[ 225], 10.00th=[ 233], 20.00th=[ 251], | 30.00th=[ 265], 40.00th=[ 277], 50.00th=[ 289], 60.00th=[ 310], | 70.00th=[ 326], 80.00th=[ 351], 90.00th=[ 396], 95.00th=[ 457], | 99.00th=[ 1647], 99.50th=[ 4752], 99.90th=[17171], 99.95th=[21627], | 99.99th=[29492] bw ( KiB/s): min=145024, max=185984, per=99.93%, avg=166705.17, stdev=8147.93, samples=125 iops : min= 2266, max= 2906, avg=2604.73, stdev=127.30, samples=125 lat (usec) : 250=19.99%, 500=76.44%, 750=1.75%, 1000=0.39% lat (msec) : 2=0.55%, 4=0.32%, 10=0.30%, 20=0.19%, 50=0.07% cpu : usr=1.12%, sys=51.92%, ctx=356912, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=163MiB/s (171MB/s), 163MiB/s-163MiB/s (171MB/s-171MB/s), io=10.0GiB (10.7GB), run=62853-62853msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64 0.82s user 32.94s system 53% cpu 1:03.27 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=38.6MiB/s][r=0,w=9869 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=20004: Sun May 27 20:26:27 2018 write: IOPS=8342, BW=32.6MiB/s (34.2MB/s)(10.0GiB/314232msec) clat (usec): min=50, max=105756, avg=118.08, stdev=324.34 lat (usec): min=50, max=105757, avg=118.32, stdev=324.38 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 67], 10.00th=[ 70], 20.00th=[ 74], | 30.00th=[ 78], 40.00th=[ 83], 50.00th=[ 87], 60.00th=[ 92], | 70.00th=[ 97], 80.00th=[ 104], 90.00th=[ 163], 95.00th=[ 206], | 99.00th=[ 355], 99.50th=[ 1004], 99.90th=[ 5342], 99.95th=[ 7701], | 99.99th=[11600] bw ( KiB/s): min=27041, max=43760, per=99.93%, avg=33346.93, stdev=3265.16, samples=628 iops : min= 6760, max=10940, avg=8336.70, stdev=816.29, samples=628 lat (usec) : 100=75.71%, 250=22.00%, 500=1.45%, 750=0.20%, 1000=0.13% lat (msec) : 2=0.20%, 4=0.15%, 10=0.12%, 20=0.02%, 50=0.01% lat (msec) : 250=0.01% cpu : usr=2.46%, sys=30.54%, ctx=5281377, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=32.6MiB/s (34.2MB/s), 32.6MiB/s-32.6MiB/s (34.2MB/s-34.2MB/s), io=10.0GiB (10.7GB), run=314232-314232msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k 8.05s user 96.59s system 33% cpu 5:14.67 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=6896KiB/s][r=0,w=13.8k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=6507: Sun May 27 18:12:21 2018 write: IOPS=7214, BW=3607KiB/s (3694kB/s)(10.0GiB/2906746msec) clat (usec): min=45, max=305426, avg=136.53, stdev=392.90 lat (usec): min=45, max=305426, avg=136.74, stdev=392.94 clat percentiles (usec): | 1.00th=[ 61], 5.00th=[ 66], 10.00th=[ 69], 20.00th=[ 74], | 30.00th=[ 79], 40.00th=[ 86], 50.00th=[ 91], 60.00th=[ 98], | 70.00th=[ 117], 80.00th=[ 155], 90.00th=[ 196], 95.00th=[ 227], | 99.00th=[ 523], 99.50th=[ 1516], 99.90th=[ 6390], 99.95th=[ 9241], | 99.99th=[13435] bw ( KiB/s): min= 1402, max= 7409, per=99.96%, avg=3605.60, stdev=310.67, samples=5813 iops : min= 2804, max=14818, avg=7211.24, stdev=621.35, samples=5813 lat (usec) : 50=0.01%, 100=62.83%, 250=34.15%, 500=2.00%, 750=0.18% lat (usec) : 1000=0.12% lat (msec) : 2=0.34%, 4=0.19%, 10=0.15%, 20=0.04%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=2.42%, sys=27.94%, ctx=42830371, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=3607KiB/s (3694kB/s), 3607KiB/s-3607KiB/s (3694kB/s-3694kB/s), io=10.0GiB (10.7GB), run=2906746-2906746msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=51 72.66s user 816.32s system 30% cpu 48:27.42 total zpool create -o ashift=12 myzpool /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN zfs create myzpool/mydataset zfs set recordsize=16K myzpool/mydataset # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=132MiB/s,w=0KiB/s][r=270k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=15887: Sun May 27 20:58:45 2018 read: IOPS=142k, BW=69.1MiB/s (72.5MB/s)(10.0GiB/148186msec) clat (nsec): min=1480, max=360272, avg=6553.78, stdev=1529.27 lat (nsec): min=1510, max=360302, avg=6588.08, stdev=1530.40 clat percentiles (nsec): | 1.00th=[ 2008], 5.00th=[ 3120], 10.00th=[ 5664], 20.00th=[ 5920], | 30.00th=[ 6112], 40.00th=[ 6304], 50.00th=[ 6496], 60.00th=[ 6688], | 70.00th=[ 6944], 80.00th=[ 7456], 90.00th=[ 8096], 95.00th=[ 8384], | 99.00th=[ 9024], 99.50th=[ 9408], 99.90th=[22144], 99.95th=[27520], | 99.99th=[34048] bw ( KiB/s): min=66553, max=148552, per=99.77%, avg=70595.82, stdev=7402.84, samples=296 iops : min=133106, max=297104, avg=141191.88, stdev=14805.69, samples=296 lat (usec) : 2=0.99%, 4=4.23%, 10=94.47%, 20=0.14%, 50=0.16% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=8.07%, sys=91.68%, ctx=10212, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=69.1MiB/s (72.5MB/s), 69.1MiB/s-69.1MiB/s (72.5MB/s-72.5MB/s), io=10.0GiB (10.7GB), run=148186-148186msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 14.88s user 211.95s system 99% cpu 3:47.83 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=625MiB/s,w=0KiB/s][r=160k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=1919: Sun May 27 21:00:48 2018 read: IOPS=141k, BW=552MiB/s (578MB/s)(10.0GiB/18562msec) clat (nsec): min=1700, max=321267, avg=6597.32, stdev=1657.15 lat (nsec): min=1740, max=321307, avg=6633.75, stdev=1658.08 clat percentiles (nsec): | 1.00th=[ 2128], 5.00th=[ 3024], 10.00th=[ 5664], 20.00th=[ 5984], | 30.00th=[ 6240], 40.00th=[ 6368], 50.00th=[ 6560], 60.00th=[ 6752], | 70.00th=[ 7008], 80.00th=[ 7456], 90.00th=[ 8160], 95.00th=[ 8512], | 99.00th=[ 9152], 99.50th=[ 9536], 99.90th=[21632], 99.95th=[23424], | 99.99th=[38144] bw ( KiB/s): min=523824, max=731288, per=99.76%, avg=563570.00, stdev=37736.44, samples=37 iops : min=130956, max=182822, avg=140892.38, stdev=9434.12, samples=37 lat (usec) : 2=0.77%, 4=5.47%, 10=93.42%, 20=0.17%, 50=0.17% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=7.02%, sys=92.72%, ctx=433, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=552MiB/s (578MB/s), 552MiB/s-552MiB/s (578MB/s-578MB/s), io=10.0GiB (10.7GB), run=18562-18562msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k 3.06s user 34.14s system 96% cpu 38.681 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=2681MiB/s,w=0KiB/s][r=42.9k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=17639: Sun May 27 21:02:12 2018 read: IOPS=42.6k, BW=2660MiB/s (2790MB/s)(10.0GiB/3849msec) clat (usec): min=7, max=267, avg=22.96, stdev= 4.84 lat (usec): min=8, max=267, avg=23.00, stdev= 4.85 clat percentiles (usec): | 1.00th=[ 20], 5.00th=[ 20], 10.00th=[ 21], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 22], 50.00th=[ 22], 60.00th=[ 22], | 70.00th=[ 25], 80.00th=[ 26], 90.00th=[ 27], 95.00th=[ 27], | 99.00th=[ 48], 99.50th=[ 55], 99.90th=[ 85], 99.95th=[ 87], | 99.99th=[ 117] bw ( MiB/s): min= 2550, max= 2690, per=99.94%, avg=2658.95, stdev=51.44, samples=7 iops : min=40800, max=43042, avg=42543.14, stdev=823.00, samples=7 lat (usec) : 10=0.17%, 20=5.91%, 50=93.01%, 100=0.89%, 250=0.02% lat (usec) : 500=0.01% cpu : usr=2.65%, sys=97.04%, ctx=55, majf=0, minf=25 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=2660MiB/s (2790MB/s), 2660MiB/s-2660MiB/s (2790MB/s-2790MB/s), io=10.0GiB (10.7GB), run=3849-3849msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k 1.92s user 11.26s system 95% cpu 13.862 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=3391MiB/s,w=0KiB/s][r=3391,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=1747: Sun May 27 21:04:47 2018 read: IOPS=3335, BW=3336MiB/s (3498MB/s)(10.0GiB/3070msec) clat (usec): min=99, max=658, avg=299.03, stdev=38.00 lat (usec): min=99, max=658, avg=299.07, stdev=38.00 clat percentiles (usec): | 1.00th=[ 251], 5.00th=[ 255], 10.00th=[ 258], 20.00th=[ 265], | 30.00th=[ 277], 40.00th=[ 285], 50.00th=[ 289], 60.00th=[ 293], | 70.00th=[ 322], 80.00th=[ 334], 90.00th=[ 359], 95.00th=[ 367], | 99.00th=[ 375], 99.50th=[ 379], 99.90th=[ 461], 99.95th=[ 478], | 99.99th=[ 652] bw ( MiB/s): min= 3283, max= 3420, per=99.86%, avg=3330.91, stdev=50.35, samples=6 iops : min= 3283, max= 3420, avg=3330.83, stdev=50.43, samples=6 lat (usec) : 100=0.01%, 250=0.97%, 500=98.97%, 750=0.05% cpu : usr=0.29%, sys=99.48%, ctx=8, majf=0, minf=266 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=3336MiB/s (3498MB/s), 3336MiB/s-3336MiB/s (3498MB/s-3498MB/s), io=10.0GiB (10.7GB), run=3070-3070msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m 1.83s user 10.59s system 94% cpu 13.117 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [R(1)][100.0%][r=2979MiB/s,w=0KiB/s][r=2978,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=16199: Sun May 27 21:06:20 2018 read: IOPS=2947, BW=2948MiB/s (3091MB/s)(10.0GiB/3474msec) clat (usec): min=89, max=917, avg=338.73, stdev=48.27 lat (usec): min=89, max=917, avg=338.78, stdev=48.27 clat percentiles (usec): | 1.00th=[ 293], 5.00th=[ 297], 10.00th=[ 297], 20.00th=[ 302], | 30.00th=[ 306], 40.00th=[ 310], 50.00th=[ 314], 60.00th=[ 326], | 70.00th=[ 359], 80.00th=[ 388], 90.00th=[ 412], 95.00th=[ 420], | 99.00th=[ 461], 99.50th=[ 465], 99.90th=[ 578], 99.95th=[ 693], | 99.99th=[ 807] bw ( MiB/s): min= 2818, max= 2982, per=99.71%, avg=2939.01, stdev=65.45, samples=6 iops : min= 2818, max= 2982, avg=2939.00, stdev=65.44, samples=6 lat (usec) : 100=0.12%, 250=0.05%, 500=99.64%, 750=0.18%, 1000=0.02% cpu : usr=0.00%, sys=99.83%, ctx=7, majf=0, minf=266 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=2948MiB/s (3091MB/s), 2948MiB/s-2948MiB/s (3091MB/s-3091MB/s), io=10.0GiB (10.7GB), run=3474-3474msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m 1.93s user 10.82s system 93% cpu 13.565 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=543MiB/s][r=0,w=542 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=16393: Sun May 27 21:08:37 2018 write: IOPS=537, BW=538MiB/s (564MB/s)(10.0GiB/19039msec) clat (usec): min=1248, max=21480, avg=1828.89, stdev=518.00 lat (usec): min=1264, max=21529, avg=1856.89, stdev=522.17 clat percentiles (usec): | 1.00th=[ 1319], 5.00th=[ 1352], 10.00th=[ 1369], 20.00th=[ 1401], | 30.00th=[ 1434], 40.00th=[ 1549], 50.00th=[ 1680], 60.00th=[ 1909], | 70.00th=[ 2024], 80.00th=[ 2180], 90.00th=[ 2474], 95.00th=[ 2704], | 99.00th=[ 3294], 99.50th=[ 3621], 99.90th=[ 4359], 99.95th=[ 4621], | 99.99th=[ 8291] bw ( KiB/s): min=530432, max=577536, per=100.00%, avg=550829.89, stdev=11057.26, samples=38 iops : min= 518, max= 564, avg=537.89, stdev=10.83, samples=38 lat (msec) : 2=68.35%, 4=31.40%, 10=0.24%, 50=0.01% cpu : usr=1.55%, sys=58.53%, ctx=69707, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=538MiB/s (564MB/s), 538MiB/s-538MiB/s (564MB/s-564MB/s), io=10.0GiB (10.7GB), run=19039-19039msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m 0.38s user 11.42s system 60% cpu 19.439 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=493MiB/s][r=0,w=493 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=30181: Sun May 27 21:11:38 2018 write: IOPS=518, BW=519MiB/s (544MB/s)(10.0GiB/19733msec) clat (usec): min=1292, max=104789, avg=1895.50, stdev=1525.36 lat (usec): min=1309, max=104833, avg=1923.40, stdev=1527.00 clat percentiles (usec): | 1.00th=[ 1369], 5.00th=[ 1401], 10.00th=[ 1434], 20.00th=[ 1467], | 30.00th=[ 1500], 40.00th=[ 1549], 50.00th=[ 1729], 60.00th=[ 1942], | 70.00th=[ 2073], 80.00th=[ 2245], 90.00th=[ 2507], 95.00th=[ 2802], | 99.00th=[ 3359], 99.50th=[ 3654], 99.90th=[ 4424], 99.95th=[ 7439], | 99.99th=[103285] bw ( KiB/s): min=425984, max=555008, per=99.98%, avg=531294.28, stdev=25116.03, samples=39 iops : min= 416, max= 542, avg=518.77, stdev=24.51, samples=39 lat (msec) : 2=64.34%, 4=35.44%, 10=0.19%, 20=0.01%, 50=0.01% lat (msec) : 250=0.02% cpu : usr=1.65%, sys=59.90%, ctx=50441, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=519MiB/s (544MB/s), 519MiB/s-519MiB/s (544MB/s-544MB/s), io=10.0GiB (10.7GB), run=19733-19733msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m 0.44s user 12.10s system 62% cpu 20.154 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=290MiB/s][r=0,w=4636 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=11602: Sun May 27 21:13:45 2018 write: IOPS=4546, BW=284MiB/s (298MB/s)(10.0GiB/36038msec) clat (usec): min=121, max=10329, avg=216.64, stdev=117.03 lat (usec): min=122, max=10331, avg=218.16, stdev=117.32 clat percentiles (usec): | 1.00th=[ 139], 5.00th=[ 147], 10.00th=[ 151], 20.00th=[ 159], | 30.00th=[ 169], 40.00th=[ 180], 50.00th=[ 190], 60.00th=[ 202], | 70.00th=[ 219], 80.00th=[ 251], 90.00th=[ 322], 95.00th=[ 367], | 99.00th=[ 529], 99.50th=[ 693], 99.90th=[ 1188], 99.95th=[ 1483], | 99.99th=[ 3523] bw ( KiB/s): min=266112, max=318080, per=99.98%, avg=290902.50, stdev=11702.90, samples=72 iops : min= 4158, max= 4970, avg=4545.33, stdev=182.85, samples=72 lat (usec) : 250=79.68%, 500=19.08%, 750=0.83%, 1000=0.23% lat (msec) : 2=0.15%, 4=0.02%, 10=0.01%, 20=0.01% cpu : usr=2.24%, sys=39.37%, ctx=337946, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=284MiB/s (298MB/s), 284MiB/s-284MiB/s (298MB/s-298MB/s), io=10.0GiB (10.7GB), run=36038-36038msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64 0.94s user 14.44s system 42% cpu 36.446 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=41.2MiB/s][r=0,w=10.5k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=5616: Sun May 27 21:21:06 2018 write: IOPS=8207, BW=32.1MiB/s (33.6MB/s)(10.0GiB/319397msec) clat (usec): min=49, max=38625, avg=119.88, stdev=344.96 lat (usec): min=49, max=38625, avg=120.13, stdev=344.99 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 71], 20.00th=[ 75], | 30.00th=[ 79], 40.00th=[ 84], 50.00th=[ 90], 60.00th=[ 95], | 70.00th=[ 101], 80.00th=[ 113], 90.00th=[ 174], 95.00th=[ 208], | 99.00th=[ 334], 99.50th=[ 635], 99.90th=[ 5407], 99.95th=[ 9110], | 99.99th=[13698] bw ( KiB/s): min=28496, max=42160, per=99.90%, avg=32796.64, stdev=2293.64, samples=638 iops : min= 7124, max=10540, avg=8199.13, stdev=573.41, samples=638 lat (usec) : 50=0.01%, 100=68.87%, 250=29.04%, 500=1.47%, 750=0.17% lat (usec) : 1000=0.08% lat (msec) : 2=0.15%, 4=0.09%, 10=0.09%, 20=0.04%, 50=0.01% cpu : usr=2.71%, sys=32.80%, ctx=5338086, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=32.1MiB/s (33.6MB/s), 32.1MiB/s-32.1MiB/s (33.6MB/s-33.6MB/s), io=10.0GiB (10.7GB), run=319397-319397msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k 9.01s user 105.35s system 35% cpu 5:19.83 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=6962KiB/s][r=0,w=13.9k IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=10869: Sun May 27 22:06:30 2018 write: IOPS=7968, BW=3984KiB/s (4080kB/s)(10.0GiB/2631712msec) clat (usec): min=40, max=92137, avg=123.58, stdev=383.15 lat (usec): min=40, max=92137, avg=123.75, stdev=383.20 clat percentiles (usec): | 1.00th=[ 59], 5.00th=[ 63], 10.00th=[ 66], 20.00th=[ 70], | 30.00th=[ 75], 40.00th=[ 80], 50.00th=[ 86], 60.00th=[ 90], | 70.00th=[ 96], 80.00th=[ 117], 90.00th=[ 200], 95.00th=[ 251], | 99.00th=[ 371], 99.50th=[ 791], 99.90th=[ 5932], 99.95th=[ 9896], | 99.99th=[14484] bw ( KiB/s): min= 3036, max= 7809, per=99.96%, avg=3982.37, stdev=237.27, samples=5263 iops : min= 6072, max=15618, avg=7964.79, stdev=474.55, samples=5263 lat (usec) : 50=0.01%, 100=75.06%, 250=19.85%, 500=4.36%, 750=0.21% lat (usec) : 1000=0.09% lat (msec) : 2=0.17%, 4=0.11%, 10=0.10%, 20=0.05%, 50=0.01% lat (msec) : 100=0.01% cpu : usr=2.57%, sys=30.38%, ctx=42430582, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=3984KiB/s (4080kB/s), 3984KiB/s-3984KiB/s (4080kB/s-4080kB/s), io=10.0GiB (10.7GB), run=2631712-2631712msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=51 69.94s user 803.06s system 33% cpu 43:52.31 total zpool create -o ashift=12 myzpool /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN zfs create myzpool/mydataset zfs set recordsize=128K myzpool/mydataset # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][99.6%][r=46.4MiB/s,w=0KiB/s][r=95.1k,w=0 IOPS][eta 00m:02s] temp: (groupid=0, jobs=1): err= 0: pid=11009: Mon May 28 08:33:56 2018 read: IOPS=47.2k, BW=23.0MiB/s (24.2MB/s)(10.0GiB/444544msec) clat (nsec): min=1460, max=100112k, avg=20635.45, stdev=22317.38 lat (nsec): min=1500, max=100112k, avg=20673.93, stdev=22317.50 clat percentiles (nsec): | 1.00th=[ 2416], 5.00th=[18816], 10.00th=[19328], 20.00th=[19840], | 30.00th=[20352], 40.00th=[20608], 50.00th=[20864], 60.00th=[21120], | 70.00th=[21376], 80.00th=[21888], 90.00th=[22656], 95.00th=[23168], | 99.00th=[24960], 99.50th=[39168], 99.90th=[43264], 99.95th=[44288], | 99.99th=[47872] bw ( KiB/s): min=18488, max=89613, per=99.90%, avg=23564.41, stdev=2789.04, samples=889 iops : min=36976, max=179226, avg=47128.93, stdev=5578.09, samples=889 lat (usec) : 2=0.61%, 4=1.83%, 10=0.01%, 20=20.12%, 50=77.43% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 4=0.01%, 250=0.01% cpu : usr=3.01%, sys=96.63%, ctx=1209, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=23.0MiB/s (24.2MB/s), 23.0MiB/s-23.0MiB/s (24.2MB/s-24.2MB/s), io=10.0GiB (10.7GB), run=444544-444544msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 16.21s user 500.82s system 99% cpu 8:38.50 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][97.8%][r=6678KiB/s,w=0KiB/s][r=1669,w=0 IOPS][eta 00m:04s] temp: (groupid=0, jobs=1): err= 0: pid=12719: Mon May 28 08:40:15 2018 read: IOPS=14.6k, BW=57.0MiB/s (59.8MB/s)(10.0GiB/179550msec) clat (nsec): min=1580, max=3999.5M, avg=67942.66, stdev=11122494.44 lat (nsec): min=1610, max=3999.5M, avg=67980.01, stdev=11122494.46 clat percentiles (usec): | 1.00th=[ 3], 5.00th=[ 4], 10.00th=[ 20], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 22], | 70.00th=[ 22], 80.00th=[ 23], 90.00th=[ 23], 95.00th=[ 24], | 99.00th=[ 27], 99.50th=[ 39], 99.90th=[ 46], 99.95th=[ 47], | 99.99th=[ 101] bw ( KiB/s): min= 8, max=400552, per=100.00%, avg=117352.38, stdev=77696.75, samples=178 iops : min= 2, max=100138, avg=29338.04, stdev=19424.18, samples=178 lat (usec) : 2=0.84%, 4=4.35%, 10=0.03%, 20=12.99%, 50=81.76% lat (usec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 4=0.01%, 10=0.01%, 100=0.01%, 250=0.01%, 500=0.01% lat (msec) : 750=0.01%, 1000=0.01% cpu : usr=2.94%, sys=96.95%, ctx=511, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=57.0MiB/s (59.8MB/s), 57.0MiB/s-57.0MiB/s (59.8MB/s-59.8MB/s), io=10.0GiB (10.7GB), run=179550-179550msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k 6.99s user 186.67s system 99% cpu 3:14.01 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=2826MiB/s,w=0KiB/s][r=45.2k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=6743: Mon May 28 08:46:36 2018 read: IOPS=44.8k, BW=2799MiB/s (2935MB/s)(10.0GiB/3659msec) clat (usec): min=3, max=198, avg=21.81, stdev= 4.09 lat (usec): min=3, max=198, avg=21.85, stdev= 4.09 clat percentiles (nsec): | 1.00th=[ 4704], 5.00th=[19584], 10.00th=[20352], 20.00th=[20864], | 30.00th=[21376], 40.00th=[21632], 50.00th=[21888], 60.00th=[22400], | 70.00th=[22656], 80.00th=[23168], 90.00th=[24192], 95.00th=[25216], | 99.00th=[33024], 99.50th=[39168], 99.90th=[44288], 99.95th=[55552], | 99.99th=[89600] bw ( MiB/s): min= 2515, max= 2941, per=99.35%, avg=2780.34, stdev=129.59, samples=7 iops : min=40246, max=47066, avg=44485.71, stdev=2073.65, samples=7 lat (usec) : 4=0.86%, 10=2.91%, 20=3.03%, 50=93.14%, 100=0.05% lat (usec) : 250=0.01% cpu : usr=2.79%, sys=96.91%, ctx=40, majf=0, minf=24 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=2799MiB/s (2935MB/s), 2799MiB/s-2799MiB/s (2935MB/s-2935MB/s), io=10.0GiB (10.7GB), run=3659-3659msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k 1.72s user 7.88s system 71% cpu 13.460 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1) temp: (groupid=0, jobs=1): err= 0: pid=22748: Mon May 28 08:53:52 2018 read: IOPS=7062, BW=7062MiB/s (7405MB/s)(10.0GiB/1450msec) clat (usec): min=123, max=676, avg=140.91, stdev=16.53 lat (usec): min=123, max=676, avg=140.95, stdev=16.54 clat percentiles (usec): | 1.00th=[ 130], 5.00th=[ 133], 10.00th=[ 133], 20.00th=[ 135], | 30.00th=[ 137], 40.00th=[ 137], 50.00th=[ 137], 60.00th=[ 139], | 70.00th=[ 141], 80.00th=[ 143], 90.00th=[ 151], 95.00th=[ 165], | 99.00th=[ 206], 99.50th=[ 239], 99.90th=[ 314], 99.95th=[ 318], | 99.99th=[ 594] bw ( MiB/s): min= 6830, max= 7142, per=98.92%, avg=6986.00, stdev=220.62, samples=2 iops : min= 6830, max= 7142, avg=6986.00, stdev=220.62, samples=2 lat (usec) : 250=99.73%, 500=0.25%, 750=0.02% cpu : usr=0.21%, sys=99.59%, ctx=12, majf=0, minf=265 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=7062MiB/s (7405MB/s), 7062MiB/s-7062MiB/s (7405MB/s-7405MB/s), io=10.0GiB (10.7GB), run=1450-1450msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m 1.99s user 4.65s system 65% cpu 10.130 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1) temp: (groupid=0, jobs=1): err= 0: pid=11462: Mon May 28 08:59:39 2018 read: IOPS=6320, BW=6321MiB/s (6628MB/s)(10.0GiB/1620msec) clat (usec): min=129, max=858, avg=157.63, stdev=27.76 lat (usec): min=129, max=859, avg=157.70, stdev=27.77 clat percentiles (usec): | 1.00th=[ 135], 5.00th=[ 137], 10.00th=[ 137], 20.00th=[ 139], | 30.00th=[ 141], 40.00th=[ 143], 50.00th=[ 149], 60.00th=[ 151], | 70.00th=[ 155], 80.00th=[ 174], 90.00th=[ 206], 95.00th=[ 215], | 99.00th=[ 239], 99.50th=[ 255], 99.90th=[ 277], 99.95th=[ 289], | 99.99th=[ 306] bw ( MiB/s): min= 5068, max= 7032, per=99.21%, avg=6271.33, stdev=1054.18, samples=3 iops : min= 5068, max= 7032, avg=6271.33, stdev=1054.18, samples=3 lat (usec) : 250=99.32%, 500=0.67%, 1000=0.01% cpu : usr=0.19%, sys=99.69%, ctx=5, majf=0, minf=265 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=6321MiB/s (6628MB/s), 6321MiB/s-6321MiB/s (6628MB/s-6628MB/s), io=10.0GiB (10.7GB), run=1620-1620msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m 1.86s user 4.84s system 68% cpu 9.775 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=933MiB/s][r=0,w=933 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=12977: Mon May 28 09:00:34 2018 write: IOPS=932, BW=933MiB/s (978MB/s)(10.0GiB/10977msec) clat (usec): min=907, max=1631, avg=1047.87, stdev=61.35 lat (usec): min=918, max=1638, avg=1070.15, stdev=63.02 clat percentiles (usec): | 1.00th=[ 938], 5.00th=[ 963], 10.00th=[ 979], 20.00th=[ 996], | 30.00th=[ 1012], 40.00th=[ 1029], 50.00th=[ 1045], 60.00th=[ 1057], | 70.00th=[ 1074], 80.00th=[ 1090], 90.00th=[ 1123], 95.00th=[ 1156], | 99.00th=[ 1237], 99.50th=[ 1287], 99.90th=[ 1467], 99.95th=[ 1516], | 99.99th=[ 1565] bw ( KiB/s): min=942080, max=966656, per=100.00%, avg=955447.57, stdev=5476.82, samples=21 iops : min= 920, max= 944, avg=933.05, stdev= 5.35, samples=21 lat (usec) : 1000=21.31% lat (msec) : 2=78.69% cpu : usr=2.88%, sys=28.69%, ctx=21735, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=933MiB/s (978MB/s), 933MiB/s-933MiB/s (978MB/s-978MB/s), io=10.0GiB (10.7GB), run=10977-10977msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=916MiB/s][r=0,w=915 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=18133: Mon May 28 09:01:56 2018 write: IOPS=908, BW=909MiB/s (953MB/s)(10.0GiB/11267msec) clat (usec): min=908, max=101227, avg=1074.57, stdev=993.05 lat (usec): min=927, max=101242, avg=1097.60, stdev=993.20 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 971], 10.00th=[ 979], 20.00th=[ 1004], | 30.00th=[ 1029], 40.00th=[ 1045], 50.00th=[ 1057], 60.00th=[ 1074], | 70.00th=[ 1090], 80.00th=[ 1106], 90.00th=[ 1139], 95.00th=[ 1188], | 99.00th=[ 1369], 99.50th=[ 1434], 99.90th=[ 1582], 99.95th=[ 1745], | 99.99th=[ 2376] bw ( KiB/s): min=755712, max=950272, per=99.95%, avg=930164.36, stdev=39392.22, samples=22 iops : min= 738, max= 928, avg=908.36, stdev=38.47, samples=22 lat (usec) : 1000=17.99% lat (msec) : 2=81.97%, 4=0.03%, 250=0.01% cpu : usr=2.65%, sys=29.35%, ctx=21301, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=909MiB/s (953MB/s), 909MiB/s-909MiB/s (953MB/s-953MB/s), io=10.0GiB (10.7GB), run=11267-11267msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=252MiB/s][r=0,w=4024 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=32050: Mon May 28 09:04:49 2018 write: IOPS=4036, BW=252MiB/s (265MB/s)(10.0GiB/40588msec) clat (usec): min=160, max=3025, avg=244.47, stdev=62.79 lat (usec): min=161, max=3027, avg=245.84, stdev=62.95 clat percentiles (usec): | 1.00th=[ 186], 5.00th=[ 194], 10.00th=[ 202], 20.00th=[ 215], | 30.00th=[ 223], 40.00th=[ 233], 50.00th=[ 239], 60.00th=[ 245], | 70.00th=[ 253], 80.00th=[ 265], 90.00th=[ 277], 95.00th=[ 297], | 99.00th=[ 412], 99.50th=[ 578], 99.90th=[ 1090], 99.95th=[ 1237], | 99.99th=[ 1860] bw ( KiB/s): min=247296, max=273664, per=99.99%, avg=258322.32, stdev=5132.74, samples=81 iops : min= 3864, max= 4276, avg=4036.25, stdev=80.17, samples=81 lat (usec) : 250=66.83%, 500=32.55%, 750=0.31%, 1000=0.16% lat (msec) : 2=0.15%, 4=0.01% cpu : usr=2.13%, sys=26.37%, ctx=333419, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=252MiB/s (265MB/s), 252MiB/s-252MiB/s (265MB/s-265MB/s), io=10.0GiB (10.7GB), run=40588-40588msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64 0.96s user 11.00s system 29% cpu 41.018 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=39.4MiB/s][r=0,w=10.1k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=8298: Mon May 28 09:12:47 2018 write: IOPS=6183, BW=24.2MiB/s (25.3MB/s)(10.0GiB/423941msec) clat (usec): min=46, max=87187, avg=159.40, stdev=168.97 lat (usec): min=46, max=87187, avg=159.73, stdev=169.02 clat percentiles (usec): | 1.00th=[ 72], 5.00th=[ 82], 10.00th=[ 86], 20.00th=[ 92], | 30.00th=[ 100], 40.00th=[ 109], 50.00th=[ 118], 60.00th=[ 159], | 70.00th=[ 194], 80.00th=[ 237], 90.00th=[ 258], 95.00th=[ 273], | 99.00th=[ 338], 99.50th=[ 553], 99.90th=[ 2180], 99.95th=[ 2835], | 99.99th=[ 4359] bw ( KiB/s): min=20760, max=40080, per=99.92%, avg=24713.14, stdev=1404.66, samples=847 iops : min= 5190, max=10020, avg=6178.25, stdev=351.17, samples=847 lat (usec) : 50=0.01%, 100=30.09%, 250=57.19%, 500=12.17%, 750=0.17% lat (usec) : 1000=0.15% lat (msec) : 2=0.11%, 4=0.10%, 10=0.02%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01% cpu : usr=2.40%, sys=34.78%, ctx=5300860, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=24.2MiB/s (25.3MB/s), 24.2MiB/s-24.2MiB/s (25.3MB/s-25.3MB/s), io=10.0GiB (10.7GB), run=423941-423941msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k 10.69s user 148.40s system 37% cpu 7:04.39 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=6179KiB/s][r=0,w=12.4k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=25425: Mon May 28 10:13:36 2018 write: IOPS=6306, BW=3153KiB/s (3229kB/s)(10.0GiB/3325643msec) clat (usec): min=43, max=147950, avg=156.40, stdev=173.24 lat (usec): min=43, max=147950, avg=156.59, stdev=173.27 clat percentiles (usec): | 1.00th=[ 71], 5.00th=[ 79], 10.00th=[ 82], 20.00th=[ 89], | 30.00th=[ 97], 40.00th=[ 104], 50.00th=[ 113], 60.00th=[ 157], | 70.00th=[ 192], 80.00th=[ 235], 90.00th=[ 253], 95.00th=[ 269], | 99.00th=[ 334], 99.50th=[ 545], 99.90th=[ 2212], 99.95th=[ 2900], | 99.99th=[ 4359] bw ( KiB/s): min= 2200, max= 6090, per=99.98%, avg=3152.38, stdev=158.05, samples=6651 iops : min= 4400, max=12180, avg=6304.82, stdev=316.09, samples=6651 lat (usec) : 50=0.01%, 100=34.52%, 250=53.93%, 500=11.01%, 750=0.15% lat (usec) : 1000=0.16% lat (msec) : 2=0.11%, 4=0.11%, 10=0.02%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=2.31%, sys=34.93%, ctx=42392184, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=3153KiB/s (3229kB/s), 3153KiB/s-3153KiB/s (3229kB/s-3229kB/s), io=10.0GiB (10.7GB), run=3325643-3325643msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=51 80.19s user 1167.43s system 37% cpu 55:26.32 total zpool create -o ashift=13 myzpool /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN zfs create myzpool/mydataset zfs set recordsize=128K myzpool/mydataset # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][99.6%][r=48.9MiB/s,w=0KiB/s][r=100k,w=0 IOPS][eta 00m:02s] temp: (groupid=0, jobs=1): err= 0: pid=584: Mon May 28 11:52:32 2018 read: IOPS=45.9k, BW=22.4MiB/s (23.5MB/s)(10.0GiB/456397msec) clat (nsec): min=1420, max=100746k, avg=21151.16, stdev=31775.46 lat (nsec): min=1460, max=100746k, avg=21185.86, stdev=31775.64 clat percentiles (usec): | 1.00th=[ 3], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 21], | 30.00th=[ 21], 40.00th=[ 21], 50.00th=[ 22], 60.00th=[ 22], | 70.00th=[ 23], 80.00th=[ 23], 90.00th=[ 24], 95.00th=[ 24], | 99.00th=[ 27], 99.50th=[ 41], 99.90th=[ 47], 99.95th=[ 48], | 99.99th=[ 169] bw ( KiB/s): min=16749, max=61314, per=99.64%, avg=22892.12, stdev=2062.43, samples=912 iops : min=33498, max=122628, avg=45784.44, stdev=4124.87, samples=912 lat (usec) : 2=0.56%, 4=1.87%, 10=0.01%, 20=14.97%, 50=82.57% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 250=0.01% cpu : usr=3.15%, sys=96.25%, ctx=3887, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=22.4MiB/s (23.5MB/s), 22.4MiB/s-22.4MiB/s (23.5MB/s-23.5MB/s), io=10.0GiB (10.7GB), run=456397-456397msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 17.26s user 510.52s system 99% cpu 8:50.37 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=371MiB/s,w=0KiB/s][r=94.0k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=32676: Mon May 28 11:41:59 2018 read: IOPS=47.6k, BW=186MiB/s (195MB/s)(10.0GiB/55042msec) clat (nsec): min=1530, max=2913.9k, avg=20447.23, stdev=4970.06 lat (nsec): min=1570, max=2916.7k, avg=20483.10, stdev=4971.25 clat percentiles (nsec): | 1.00th=[ 2096], 5.00th=[ 3376], 10.00th=[19584], 20.00th=[20096], | 30.00th=[20608], 40.00th=[20864], 50.00th=[21120], 60.00th=[21376], | 70.00th=[21632], 80.00th=[22144], 90.00th=[22912], 95.00th=[23680], | 99.00th=[25984], 99.50th=[38144], 99.90th=[43264], 99.95th=[44288], | 99.99th=[47360] bw ( KiB/s): min=171496, max=463024, per=99.72%, avg=189966.99, stdev=30271.35, samples=110 iops : min=42874, max=115756, avg=47491.72, stdev=7567.81, samples=110 lat (usec) : 2=0.78%, 4=4.44%, 10=0.01%, 20=12.38%, 50=82.39% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01% lat (msec) : 4=0.01% cpu : usr=2.73%, sys=97.01%, ctx=104, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=186MiB/s (195MB/s), 186MiB/s-186MiB/s (195MB/s-195MB/s), io=10.0GiB (10.7GB), run=55042-55042msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k 3.12s user 66.03s system 99% cpu 1:09.60 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=2880MiB/s,w=0KiB/s][r=46.1k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=1289: Mon May 28 11:39:57 2018 read: IOPS=45.3k, BW=2829MiB/s (2966MB/s)(10.0GiB/3620msec) clat (usec): min=3, max=122, avg=21.56, stdev= 3.82 lat (usec): min=3, max=123, avg=21.60, stdev= 3.82 clat percentiles (nsec): | 1.00th=[ 4896], 5.00th=[19328], 10.00th=[20096], 20.00th=[20608], | 30.00th=[21120], 40.00th=[21376], 50.00th=[21632], 60.00th=[22144], | 70.00th=[22400], 80.00th=[23168], 90.00th=[23936], 95.00th=[24960], | 99.00th=[27776], 99.50th=[39680], 99.90th=[46336], 99.95th=[49408], | 99.99th=[75264] bw ( MiB/s): min= 2528, max= 3017, per=99.40%, avg=2811.68, stdev=154.30, samples=7 iops : min=40460, max=48277, avg=44986.43, stdev=2468.31, samples=7 lat (usec) : 4=0.87%, 10=2.82%, 20=4.90%, 50=91.37%, 100=0.04% lat (usec) : 250=0.01% cpu : usr=2.18%, sys=97.51%, ctx=7, majf=0, minf=24 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=2829MiB/s (2966MB/s), 2829MiB/s-2829MiB/s (2966MB/s-2966MB/s), io=10.0GiB (10.7GB), run=3620-3620msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k 1.53s user 8.14s system 70% cpu 13.645 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1) temp: (groupid=0, jobs=1): err= 0: pid=7684: Mon May 28 11:39:02 2018 read: IOPS=4614, BW=4615MiB/s (4839MB/s)(10.0GiB/2219msec) clat (usec): min=104, max=783164, avg=216.05, stdev=7737.95 lat (usec): min=104, max=783164, avg=216.09, stdev=7737.95 clat percentiles (usec): | 1.00th=[ 131], 5.00th=[ 133], 10.00th=[ 135], 20.00th=[ 135], | 30.00th=[ 137], 40.00th=[ 137], 50.00th=[ 139], 60.00th=[ 139], | 70.00th=[ 141], 80.00th=[ 143], 90.00th=[ 149], 95.00th=[ 155], | 99.00th=[ 169], 99.50th=[ 178], 99.90th=[ 186], 99.95th=[ 190], | 99.99th=[ 506] bw ( MiB/s): min= 604, max= 7160, per=93.92%, avg=4334.00, stdev=3304.51, samples=4 iops : min= 604, max= 7160, avg=4334.00, stdev=3304.51, samples=4 lat (usec) : 250=99.98%, 750=0.01% lat (msec) : 1000=0.01% cpu : usr=0.68%, sys=99.14%, ctx=8, majf=0, minf=265 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=4615MiB/s (4839MB/s), 4615MiB/s-4615MiB/s (4839MB/s-4839MB/s), io=10.0GiB (10.7GB), run=2219-2219msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m 1.89s user 5.51s system 67% cpu 10.926 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1) temp: (groupid=0, jobs=1): err= 0: pid=18077: Mon May 28 11:38:08 2018 read: IOPS=6589, BW=6589MiB/s (6910MB/s)(10.0GiB/1554msec) clat (usec): min=132, max=520, avg=151.21, stdev= 9.48 lat (usec): min=132, max=520, avg=151.26, stdev= 9.49 clat percentiles (usec): | 1.00th=[ 143], 5.00th=[ 145], 10.00th=[ 147], 20.00th=[ 147], | 30.00th=[ 149], 40.00th=[ 149], 50.00th=[ 151], 60.00th=[ 151], | 70.00th=[ 153], 80.00th=[ 153], 90.00th=[ 155], 95.00th=[ 159], | 99.00th=[ 192], 99.50th=[ 200], 99.90th=[ 258], 99.95th=[ 262], | 99.99th=[ 420] bw ( MiB/s): min= 6506, max= 6652, per=100.00%, avg=6590.67, stdev=75.75, samples=3 iops : min= 6506, max= 6652, avg=6590.67, stdev=75.75, samples=3 lat (usec) : 250=99.81%, 500=0.18%, 750=0.01% cpu : usr=0.39%, sys=99.29%, ctx=5, majf=0, minf=266 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=6589MiB/s (6910MB/s), 6589MiB/s-6589MiB/s (6910MB/s-6910MB/s), io=10.0GiB (10.7GB), run=1554-1554msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m 1.96s user 4.85s system 65% cpu 10.373 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=920MiB/s][r=0,w=919 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=23512: Mon May 28 11:37:04 2018 write: IOPS=929, BW=930MiB/s (975MB/s)(10.0GiB/11014msec) clat (usec): min=902, max=7660, avg=1051.01, stdev=88.17 lat (usec): min=920, max=7684, avg=1073.80, stdev=89.28 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 963], 10.00th=[ 979], 20.00th=[ 1004], | 30.00th=[ 1020], 40.00th=[ 1037], 50.00th=[ 1057], 60.00th=[ 1057], | 70.00th=[ 1074], 80.00th=[ 1090], 90.00th=[ 1123], 95.00th=[ 1139], | 99.00th=[ 1221], 99.50th=[ 1287], 99.90th=[ 1434], 99.95th=[ 1467], | 99.99th=[ 1598] bw ( KiB/s): min=940032, max=964608, per=100.00%, avg=952040.73, stdev=6423.56, samples=22 iops : min= 918, max= 942, avg=929.73, stdev= 6.27, samples=22 lat (usec) : 1000=20.11% lat (msec) : 2=79.88%, 10=0.01% cpu : usr=2.29%, sys=29.47%, ctx=21771, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=930MiB/s (975MB/s), 930MiB/s-930MiB/s (975MB/s-975MB/s), io=10.0GiB (10.7GB), run=11014-11014msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=916MiB/s][r=0,w=916 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=11930: Mon May 28 11:36:01 2018 write: IOPS=911, BW=912MiB/s (956MB/s)(10.0GiB/11230msec) clat (usec): min=909, max=2386, avg=1067.90, stdev=80.71 lat (usec): min=925, max=2405, avg=1094.04, stdev=83.74 clat percentiles (usec): | 1.00th=[ 947], 5.00th=[ 963], 10.00th=[ 979], 20.00th=[ 1004], | 30.00th=[ 1029], 40.00th=[ 1045], 50.00th=[ 1057], 60.00th=[ 1074], | 70.00th=[ 1090], 80.00th=[ 1106], 90.00th=[ 1139], 95.00th=[ 1205], | 99.00th=[ 1385], 99.50th=[ 1434], 99.90th=[ 1582], 99.95th=[ 1614], | 99.99th=[ 1893] bw ( KiB/s): min=923648, max=942080, per=99.98%, avg=933540.82, stdev=4907.61, samples=22 iops : min= 902, max= 920, avg=911.64, stdev= 4.81, samples=22 lat (usec) : 1000=16.80% lat (msec) : 2=83.19%, 4=0.01% cpu : usr=3.31%, sys=29.24%, ctx=21271, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=912MiB/s (956MB/s), 912MiB/s-912MiB/s (956MB/s-956MB/s), io=10.0GiB (10.7GB), run=11230-11230msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=250MiB/s][r=0,w=3997 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=2850: Mon May 28 11:34:52 2018 write: IOPS=4007, BW=250MiB/s (263MB/s)(10.0GiB/40881msec) clat (usec): min=164, max=2969, avg=246.16, stdev=65.54 lat (usec): min=164, max=2971, avg=247.54, stdev=65.72 clat percentiles (usec): | 1.00th=[ 188], 5.00th=[ 196], 10.00th=[ 204], 20.00th=[ 217], | 30.00th=[ 225], 40.00th=[ 233], 50.00th=[ 241], 60.00th=[ 247], | 70.00th=[ 253], 80.00th=[ 265], 90.00th=[ 281], 95.00th=[ 297], | 99.00th=[ 429], 99.50th=[ 611], 99.90th=[ 1090], 99.95th=[ 1237], | 99.99th=[ 2040] bw ( KiB/s): min=241408, max=273536, per=99.97%, avg=256417.05, stdev=6797.61, samples=81 iops : min= 3772, max= 4274, avg=4006.49, stdev=106.20, samples=81 lat (usec) : 250=65.16%, 500=34.16%, 750=0.36%, 1000=0.17% lat (msec) : 2=0.14%, 4=0.01% cpu : usr=2.26%, sys=26.61%, ctx=333077, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=250MiB/s (263MB/s), 250MiB/s-250MiB/s (263MB/s-263MB/s), io=10.0GiB (10.7GB), run=40881-40881msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64 1.03s user 11.16s system 29% cpu 41.291 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][99.8%][r=0KiB/s,w=35.9MiB/s][r=0,w=9186 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=27931: Mon May 28 11:33:06 2018 write: IOPS=5832, BW=22.8MiB/s (23.9MB/s)(10.0GiB/449491msec) clat (usec): min=55, max=102115, avg=169.16, stdev=177.90 lat (usec): min=56, max=102115, avg=169.45, stdev=177.94 clat percentiles (usec): | 1.00th=[ 80], 5.00th=[ 91], 10.00th=[ 95], 20.00th=[ 103], | 30.00th=[ 112], 40.00th=[ 120], 50.00th=[ 128], 60.00th=[ 163], | 70.00th=[ 208], 80.00th=[ 245], 90.00th=[ 265], 95.00th=[ 281], | 99.00th=[ 355], 99.50th=[ 570], 99.90th=[ 2089], 99.95th=[ 2671], | 99.99th=[ 4178] bw ( KiB/s): min=18560, max=39241, per=99.91%, avg=23306.97, stdev=1303.72, samples=898 iops : min= 4640, max= 9810, avg=5826.72, stdev=325.94, samples=898 lat (usec) : 100=16.88%, 250=66.62%, 500=15.94%, 750=0.16%, 1000=0.17% lat (msec) : 2=0.12%, 4=0.10%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=2.18%, sys=33.55%, ctx=5287900, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=22.8MiB/s (23.9MB/s), 22.8MiB/s-22.8MiB/s (23.9MB/s-23.9MB/s), io=10.0GiB (10.7GB), run=449491-449491msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k 10.29s user 151.83s system 36% cpu 7:29.92 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=5369KiB/s][r=0,w=10.7k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=5394: Mon May 28 11:24:22 2018 write: IOPS=6107, BW=3054KiB/s (3127kB/s)(10.0GiB/3433799msec) clat (usec): min=45, max=104176, avg=161.38, stdev=157.73 lat (usec): min=45, max=104176, avg=161.55, stdev=157.75 clat percentiles (usec): | 1.00th=[ 75], 5.00th=[ 82], 10.00th=[ 86], 20.00th=[ 93], | 30.00th=[ 102], 40.00th=[ 110], 50.00th=[ 118], 60.00th=[ 161], | 70.00th=[ 200], 80.00th=[ 239], 90.00th=[ 260], 95.00th=[ 277], | 99.00th=[ 347], 99.50th=[ 562], 99.90th=[ 2147], 99.95th=[ 2737], | 99.99th=[ 4293] bw ( KiB/s): min= 2411, max= 5691, per=100.00%, avg=3053.05, stdev=138.80, samples=6867 iops : min= 4822, max=11382, avg=6106.15, stdev=277.61, samples=6867 lat (usec) : 50=0.01%, 100=28.30%, 250=58.18%, 500=12.96%, 750=0.17% lat (usec) : 1000=0.17% lat (msec) : 2=0.11%, 4=0.10%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=2.38%, sys=33.81%, ctx=42342670, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=3054KiB/s (3127kB/s), 3054KiB/s-3054KiB/s (3127kB/s-3127kB/s), io=10.0GiB (10.7GB), run=3433799-3433799msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=51 85.34s user 1166.82s system 36% cpu 57:14.49 total zpool create -o ashift=12 myzpool /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN zfs create myzpool/mydataset zfs set recordsize=32K myzpool/mydataset # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][99.5%][r=84.3MiB/s,w=0KiB/s][r=173k,w=0 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=7409: Mon May 28 14:38:04 2018 read: IOPS=109k, BW=53.2MiB/s (55.8MB/s)(10.0GiB/192575msec) clat (nsec): min=1649, max=732618, avg=8674.30, stdev=2157.54 lat (nsec): min=1680, max=732708, avg=8712.19, stdev=2158.55 clat percentiles (nsec): | 1.00th=[ 2352], 5.00th=[ 7200], 10.00th=[ 7520], 20.00th=[ 7840], | 30.00th=[ 8096], 40.00th=[ 8256], 50.00th=[ 8512], 60.00th=[ 8768], | 70.00th=[ 9024], 80.00th=[ 9408], 90.00th=[10176], 95.00th=[10944], | 99.00th=[15168], 99.50th=[16512], 99.90th=[28544], 99.95th=[30080], | 99.99th=[34560] bw ( KiB/s): min=49157, max=121879, per=99.88%, avg=54381.99, stdev=4726.31, samples=385 iops : min=98314, max=243758, avg=108764.15, stdev=9452.61, samples=385 lat (usec) : 2=0.24%, 4=3.49%, 10=84.12%, 20=11.87%, 50=0.27% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=6.23%, sys=93.49%, ctx=613, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=53.2MiB/s (55.8MB/s), 53.2MiB/s-53.2MiB/s (55.8MB/s-55.8MB/s), io=10.0GiB (10.7GB), run=192575-192575msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 14.80s user 255.15s system 99% cpu 4:30.72 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=538MiB/s,w=0KiB/s][r=138k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=15104: Mon May 28 14:32:35 2018 read: IOPS=111k, BW=434MiB/s (455MB/s)(10.0GiB/23599msec) clat (nsec): min=1620, max=1019.8k, avg=8513.84, stdev=2613.86 lat (nsec): min=1660, max=1019.9k, avg=8551.53, stdev=2615.03 clat percentiles (nsec): | 1.00th=[ 2024], 5.00th=[ 2864], 10.00th=[ 7456], 20.00th=[ 7840], | 30.00th=[ 8096], 40.00th=[ 8256], 50.00th=[ 8512], 60.00th=[ 8768], | 70.00th=[ 9024], 80.00th=[ 9408], 90.00th=[10176], 95.00th=[10944], | 99.00th=[15296], 99.50th=[16768], 99.90th=[27776], 99.95th=[29568], | 99.99th=[64768] bw ( KiB/s): min=387912, max=699321, per=99.57%, avg=442410.11, stdev=48209.47, samples=47 iops : min=96978, max=174830, avg=110602.47, stdev=12052.36, samples=47 lat (usec) : 2=0.92%, 4=6.16%, 10=80.51%, 20=12.12%, 50=0.28% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (msec) : 2=0.01% cpu : usr=6.15%, sys=93.54%, ctx=496, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=434MiB/s (455MB/s), 434MiB/s-434MiB/s (455MB/s-455MB/s), io=10.0GiB (10.7GB), run=23599-23599msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k 3.17s user 37.12s system 98% cpu 40.822 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=3229MiB/s,w=0KiB/s][r=51.7k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=15654: Mon May 28 14:30:18 2018 read: IOPS=51.2k, BW=3202MiB/s (3358MB/s)(10.0GiB/3198msec) clat (usec): min=7, max=422, avg=18.96, stdev= 4.93 lat (usec): min=7, max=422, avg=19.00, stdev= 4.93 clat percentiles (usec): | 1.00th=[ 16], 5.00th=[ 17], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 19], | 70.00th=[ 19], 80.00th=[ 21], 90.00th=[ 22], 95.00th=[ 24], | 99.00th=[ 29], 99.50th=[ 38], 99.90th=[ 85], 99.95th=[ 108], | 99.99th=[ 206] bw ( MiB/s): min= 3038, max= 3232, per=99.78%, avg=3194.81, stdev=76.94, samples=6 iops : min=48608, max=51724, avg=51117.00, stdev=1231.10, samples=6 lat (usec) : 10=0.50%, 20=78.40%, 50=20.88%, 100=0.16%, 250=0.06% lat (usec) : 500=0.01% cpu : usr=3.63%, sys=95.84%, ctx=215, majf=0, minf=27 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=3202MiB/s (3358MB/s), 3202MiB/s-3202MiB/s (3358MB/s-3358MB/s), io=10.0GiB (10.7GB), run=3198-3198msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k 1.89s user 8.23s system 90% cpu 11.157 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][-.-%][r=4730MiB/s,w=0KiB/s][r=4730,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=2076: Mon May 28 14:29:03 2018 read: IOPS=4695, BW=4695MiB/s (4923MB/s)(10.0GiB/2181msec) clat (usec): min=108, max=1177, avg=212.23, stdev=28.86 lat (usec): min=108, max=1178, avg=212.28, stdev=28.87 clat percentiles (usec): | 1.00th=[ 182], 5.00th=[ 186], 10.00th=[ 188], 20.00th=[ 190], | 30.00th=[ 192], 40.00th=[ 198], 50.00th=[ 215], 60.00th=[ 219], | 70.00th=[ 221], 80.00th=[ 227], 90.00th=[ 247], 95.00th=[ 255], | 99.00th=[ 306], 99.50th=[ 359], 99.90th=[ 424], 99.95th=[ 494], | 99.99th=[ 742] bw ( MiB/s): min= 4550, max= 4780, per=99.50%, avg=4671.77, stdev=94.59, samples=4 iops : min= 4550, max= 4780, avg=4671.50, stdev=94.42, samples=4 lat (usec) : 250=91.88%, 500=8.08%, 750=0.04% lat (msec) : 2=0.01% cpu : usr=0.14%, sys=99.54%, ctx=24, majf=0, minf=265 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=4695MiB/s (4923MB/s), 4695MiB/s-4695MiB/s (4923MB/s-4923MB/s), io=10.0GiB (10.7GB), run=2181-2181msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m 2.02s user 7.01s system 88% cpu 10.151 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [R(1)][-.-%][r=4458MiB/s,w=0KiB/s][r=4458,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=23286: Mon May 28 14:27:52 2018 read: IOPS=4394, BW=4395MiB/s (4608MB/s)(10.0GiB/2330msec) clat (usec): min=77, max=1178, avg=227.00, stdev=34.67 lat (usec): min=77, max=1179, avg=227.06, stdev=34.69 clat percentiles (usec): | 1.00th=[ 200], 5.00th=[ 202], 10.00th=[ 204], 20.00th=[ 206], | 30.00th=[ 208], 40.00th=[ 210], 50.00th=[ 217], 60.00th=[ 229], | 70.00th=[ 241], 80.00th=[ 247], 90.00th=[ 258], 95.00th=[ 273], | 99.00th=[ 318], 99.50th=[ 433], 99.90th=[ 562], 99.95th=[ 578], | 99.99th=[ 644] bw ( MiB/s): min= 4256, max= 4495, per=100.00%, avg=4407.81, stdev=112.43, samples=4 iops : min= 4256, max= 4495, avg=4407.75, stdev=112.43, samples=4 lat (usec) : 100=0.59%, 250=84.66%, 500=14.50%, 750=0.24% lat (msec) : 2=0.01% cpu : usr=0.00%, sys=99.79%, ctx=40, majf=0, minf=266 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=4395MiB/s (4608MB/s), 4395MiB/s-4395MiB/s (4608MB/s-4608MB/s), io=10.0GiB (10.7GB), run=2330-2330msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m 1.96s user 7.36s system 91% cpu 10.143 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=844MiB/s][r=0,w=844 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=29749: Mon May 28 14:26:47 2018 write: IOPS=824, BW=824MiB/s (864MB/s)(10.0GiB/12424msec) clat (usec): min=1005, max=101382, avg=1187.01, stdev=1066.03 lat (usec): min=1028, max=101407, avg=1211.49, stdev=1066.12 clat percentiles (usec): | 1.00th=[ 1037], 5.00th=[ 1057], 10.00th=[ 1074], 20.00th=[ 1090], | 30.00th=[ 1106], 40.00th=[ 1123], 50.00th=[ 1156], 60.00th=[ 1172], | 70.00th=[ 1205], 80.00th=[ 1254], 90.00th=[ 1303], 95.00th=[ 1369], | 99.00th=[ 1565], 99.50th=[ 1663], 99.90th=[ 1942], 99.95th=[ 2114], | 99.99th=[39584] bw ( KiB/s): min=665600, max=870400, per=99.95%, avg=843537.00, stdev=42201.01, samples=24 iops : min= 650, max= 850, avg=823.75, stdev=41.22, samples=24 lat (msec) : 2=99.93%, 4=0.05%, 50=0.01%, 250=0.01% cpu : usr=2.69%, sys=49.29%, ctx=34342, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=824MiB/s (864MB/s), 824MiB/s-824MiB/s (864MB/s-864MB/s), io=10.0GiB (10.7GB), run=12424-12424msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m 0.39s user 6.42s system 53% cpu 12.830 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=806MiB/s][r=0,w=805 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=2304: Mon May 28 14:25:37 2018 write: IOPS=802, BW=802MiB/s (841MB/s)(10.0GiB/12762msec) clat (usec): min=1035, max=3501, avg=1221.78, stdev=129.50 lat (usec): min=1054, max=3526, avg=1243.71, stdev=131.20 clat percentiles (usec): | 1.00th=[ 1057], 5.00th=[ 1090], 10.00th=[ 1106], 20.00th=[ 1123], | 30.00th=[ 1139], 40.00th=[ 1172], 50.00th=[ 1188], 60.00th=[ 1221], | 70.00th=[ 1254], 80.00th=[ 1303], 90.00th=[ 1369], 95.00th=[ 1467], | 99.00th=[ 1663], 99.50th=[ 1762], 99.90th=[ 2114], 99.95th=[ 2245], | 99.99th=[ 2573] bw ( KiB/s): min=811008, max=831488, per=99.97%, avg=821379.72, stdev=5510.70, samples=25 iops : min= 792, max= 812, avg=802.08, stdev= 5.43, samples=25 lat (msec) : 2=99.82%, 4=0.18% cpu : usr=2.12%, sys=51.70%, ctx=28130, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=802MiB/s (841MB/s), 802MiB/s-802MiB/s (841MB/s-841MB/s), io=10.0GiB (10.7GB), run=12762-12762msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m 0.38s user 6.83s system 54% cpu 13.173 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=345MiB/s][r=0,w=5518 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=23395: Mon May 28 14:15:37 2018 write: IOPS=5475, BW=342MiB/s (359MB/s)(10.0GiB/29922msec) clat (usec): min=115, max=100291, avg=179.29, stdev=305.22 lat (usec): min=116, max=100292, avg=180.64, stdev=305.26 clat percentiles (usec): | 1.00th=[ 127], 5.00th=[ 133], 10.00th=[ 137], 20.00th=[ 145], | 30.00th=[ 153], 40.00th=[ 161], 50.00th=[ 167], 60.00th=[ 172], | 70.00th=[ 178], 80.00th=[ 184], 90.00th=[ 202], 95.00th=[ 241], | 99.00th=[ 388], 99.50th=[ 578], 99.90th=[ 2311], 99.95th=[ 4293], | 99.99th=[ 6456] bw ( KiB/s): min=261632, max=375168, per=99.89%, avg=350037.22, stdev=16590.96, samples=59 iops : min= 4088, max= 5862, avg=5469.31, stdev=259.22, samples=59 lat (usec) : 250=95.47%, 500=3.89%, 750=0.31%, 1000=0.12% lat (msec) : 2=0.11%, 4=0.06%, 10=0.05%, 50=0.01%, 250=0.01% cpu : usr=2.11%, sys=37.76%, ctx=331667, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=342MiB/s (359MB/s), 342MiB/s-342MiB/s (359MB/s-359MB/s), io=10.0GiB (10.7GB), run=29922-29922msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64 0.74s user 11.56s system 40% cpu 30.336 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][99.7%][r=0KiB/s,w=41.1MiB/s][r=0,w=10.5k IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=9341: Mon May 28 14:12:39 2018 write: IOPS=8599, BW=33.6MiB/s (35.2MB/s)(10.0GiB/304837msec) clat (usec): min=46, max=140881, avg=114.36, stdev=208.68 lat (usec): min=46, max=140881, avg=114.59, stdev=208.74 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 69], 10.00th=[ 72], 20.00th=[ 76], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 90], 60.00th=[ 96], | 70.00th=[ 102], 80.00th=[ 123], 90.00th=[ 184], 95.00th=[ 229], | 99.00th=[ 371], 99.50th=[ 553], 99.90th=[ 1696], 99.95th=[ 2409], | 99.99th=[ 7570] bw ( KiB/s): min=22744, max=42536, per=99.96%, avg=34383.15, stdev=1712.29, samples=609 iops : min= 5686, max=10634, avg=8595.77, stdev=428.07, samples=609 lat (usec) : 50=0.01%, 100=66.77%, 250=29.80%, 500=2.82%, 750=0.30% lat (usec) : 1000=0.09% lat (msec) : 2=0.15%, 4=0.05%, 10=0.02%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=2.79%, sys=35.43%, ctx=5371614, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=33.6MiB/s (35.2MB/s), 33.6MiB/s-33.6MiB/s (35.2MB/s-35.2MB/s), io=10.0GiB (10.7GB), run=304837-304837msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k 8.84s user 108.60s system 38% cpu 5:05.29 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=6356KiB/s][r=0,w=12.7k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=5275: Mon May 28 12:38:54 2018 write: IOPS=8661, BW=4331KiB/s (4435kB/s)(10.0GiB/2421194msec) clat (usec): min=42, max=175063, avg=113.55, stdev=211.07 lat (usec): min=42, max=175064, avg=113.74, stdev=211.09 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 64], 10.00th=[ 67], 20.00th=[ 71], | 30.00th=[ 75], 40.00th=[ 80], 50.00th=[ 85], 60.00th=[ 90], | 70.00th=[ 96], 80.00th=[ 119], 90.00th=[ 202], 95.00th=[ 251], | 99.00th=[ 396], 99.50th=[ 578], 99.90th=[ 1811], 99.95th=[ 2802], | 99.99th=[ 8717] bw ( KiB/s): min= 2773, max= 6597, per=100.00%, avg=4329.91, stdev=198.52, samples=4842 iops : min= 5546, max=13194, avg=8659.84, stdev=397.03, samples=4842 lat (usec) : 50=0.01%, 100=74.26%, 250=20.72%, 500=4.35%, 750=0.33% lat (usec) : 1000=0.09% lat (msec) : 2=0.15%, 4=0.05%, 10=0.03%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=2.80%, sys=33.77%, ctx=42770131, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=4331KiB/s (4435kB/s), 4331KiB/s-4331KiB/s (4435kB/s-4435kB/s), io=10.0GiB (10.7GB), run=2421194-2421194msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=51 69.62s user 820.89s system 36% cpu 40:21.70 total zpool create -o ashift=12 myzpool /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN zfs create myzpool/mydataset zfs set recordsize=64K myzpool/mydataset # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][99.3%][r=65.4MiB/s,w=0KiB/s][r=134k,w=0 IOPS][eta 00m:02s] temp: (groupid=0, jobs=1): err= 0: pid=14619: Mon May 28 16:44:05 2018 read: IOPS=72.2k, BW=35.2MiB/s (36.0MB/s)(10.0GiB/290525msec) clat (nsec): min=1430, max=1000.0M, avg=13345.76, stdev=727374.84 lat (nsec): min=1460, max=1000.0M, avg=13380.35, stdev=727374.85 clat percentiles (nsec): | 1.00th=[ 2320], 5.00th=[11072], 10.00th=[11456], 20.00th=[11712], | 30.00th=[11968], 40.00th=[12224], 50.00th=[12480], 60.00th=[12608], | 70.00th=[12992], 80.00th=[13376], 90.00th=[14144], 95.00th=[15040], | 99.00th=[19584], 99.50th=[21376], 99.90th=[33536], 99.95th=[35072], | 99.99th=[39168] bw ( KiB/s): min= 1, max=116354, per=100.00%, avg=36697.62, stdev=8480.63, samples=571 iops : min= 2, max=232708, avg=73395.33, stdev=16961.26, samples=571 lat (usec) : 2=0.72%, 4=2.22%, 10=0.02%, 20=96.19%, 50=0.85% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (msec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=4.25%, sys=95.49%, ctx=394, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=35.2MiB/s (36.0MB/s), 35.2MiB/s-35.2MiB/s (36.0MB/s-36.0MB/s), io=10.0GiB (10.7GB), run=290525-290525msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 15.08s user 351.14s system 99% cpu 6:06.99 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][97.1%][r=396MiB/s,w=0KiB/s][r=101k,w=0 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=18330: Mon May 28 16:36:55 2018 read: IOPS=78.0k, BW=305MiB/s (320MB/s)(10.0GiB/33604msec) clat (nsec): min=1569, max=471294, avg=12318.50, stdev=3277.75 lat (nsec): min=1599, max=471354, avg=12354.40, stdev=3278.53 clat percentiles (nsec): | 1.00th=[ 2008], 5.00th=[ 2896], 10.00th=[11328], 20.00th=[11712], | 30.00th=[12096], 40.00th=[12352], 50.00th=[12480], 60.00th=[12736], | 70.00th=[13120], 80.00th=[13504], 90.00th=[14272], 95.00th=[15296], | 99.00th=[19840], 99.50th=[21888], 99.90th=[34048], 99.95th=[36096], | 99.99th=[43776] bw ( KiB/s): min=276696, max=603944, per=99.44%, avg=310287.84, stdev=44357.65, samples=67 iops : min=69174, max=150986, avg=77571.94, stdev=11089.43, samples=67 lat (usec) : 2=0.97%, 4=5.57%, 10=0.03%, 20=92.45%, 50=0.97% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=4.13%, sys=95.58%, ctx=246, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=305MiB/s (320MB/s), 305MiB/s-305MiB/s (320MB/s-320MB/s), io=10.0GiB (10.7GB), run=33604-33604msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k 2.98s user 45.59s system 99% cpu 49.041 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1) temp: (groupid=0, jobs=1): err= 0: pid=14800: Mon May 28 16:33:16 2018 read: IOPS=58.5k, BW=3653MiB/s (3831MB/s)(10.0GiB/2803msec) clat (usec): min=6, max=544, avg=16.57, stdev= 4.80 lat (usec): min=6, max=544, avg=16.61, stdev= 4.81 clat percentiles (usec): | 1.00th=[ 14], 5.00th=[ 15], 10.00th=[ 15], 20.00th=[ 15], | 30.00th=[ 16], 40.00th=[ 16], 50.00th=[ 16], 60.00th=[ 16], | 70.00th=[ 17], 80.00th=[ 18], 90.00th=[ 20], 95.00th=[ 23], | 99.00th=[ 32], 99.50th=[ 36], 99.90th=[ 57], 99.95th=[ 88], | 99.99th=[ 119] bw ( MiB/s): min= 2964, max= 3843, per=99.17%, avg=3623.08, stdev=379.20, samples=5 iops : min=47432, max=61499, avg=57969.60, stdev=6066.33, samples=5 lat (usec) : 10=0.47%, 20=89.95%, 50=9.45%, 100=0.11%, 250=0.01% lat (usec) : 500=0.01%, 750=0.01% cpu : usr=4.50%, sys=94.83%, ctx=2335, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=3653MiB/s (3831MB/s), 3653MiB/s-3653MiB/s (3831MB/s-3831MB/s), io=10.0GiB (10.7GB), run=2803-2803msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k 1.92s user 6.61s system 79% cpu 10.748 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][-.-%][r=5683MiB/s,w=0KiB/s][r=5682,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=24569: Mon May 28 16:30:57 2018 read: IOPS=5137, BW=5138MiB/s (5388MB/s)(10.0GiB/1993msec) clat (usec): min=93, max=990, avg=193.66, stdev=48.71 lat (usec): min=93, max=991, avg=193.71, stdev=48.72 clat percentiles (usec): | 1.00th=[ 157], 5.00th=[ 161], 10.00th=[ 163], 20.00th=[ 165], | 30.00th=[ 167], 40.00th=[ 169], 50.00th=[ 174], 60.00th=[ 188], | 70.00th=[ 194], 80.00th=[ 204], 90.00th=[ 262], 95.00th=[ 289], | 99.00th=[ 351], 99.50th=[ 392], 99.90th=[ 775], 99.95th=[ 816], | 99.99th=[ 889] bw ( MiB/s): min= 3696, max= 5628, per=96.26%, avg=4945.66, stdev=1083.76, samples=3 iops : min= 3696, max= 5628, avg=4945.33, stdev=1083.51, samples=3 lat (usec) : 100=0.02%, 250=87.11%, 500=12.73%, 750=0.03%, 1000=0.11% cpu : usr=0.00%, sys=99.50%, ctx=82, majf=0, minf=265 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=5138MiB/s (5388MB/s), 5138MiB/s-5138MiB/s (5388MB/s-5388MB/s), io=10.0GiB (10.7GB), run=1993-1993msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m 1.87s user 5.85s system 78% cpu 9.806 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1) temp: (groupid=0, jobs=1): err= 0: pid=11025: Mon May 28 16:29:49 2018 read: IOPS=6113, BW=6113MiB/s (6410MB/s)(10.0GiB/1675msec) clat (usec): min=138, max=773, avg=163.01, stdev=13.02 lat (usec): min=138, max=773, avg=163.08, stdev=13.02 clat percentiles (usec): | 1.00th=[ 153], 5.00th=[ 155], 10.00th=[ 155], 20.00th=[ 157], | 30.00th=[ 159], 40.00th=[ 159], 50.00th=[ 161], 60.00th=[ 161], | 70.00th=[ 163], 80.00th=[ 165], 90.00th=[ 174], 95.00th=[ 180], | 99.00th=[ 210], 99.50th=[ 235], 99.90th=[ 273], 99.95th=[ 281], | 99.99th=[ 334] bw ( MiB/s): min= 5912, max= 6208, per=99.88%, avg=6106.00, stdev=168.08, samples=3 iops : min= 5912, max= 6208, avg=6106.00, stdev=168.08, samples=3 lat (usec) : 250=99.67%, 500=0.32%, 1000=0.01% cpu : usr=0.36%, sys=99.34%, ctx=3, majf=0, minf=265 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=6113MiB/s (6410MB/s), 6113MiB/s-6113MiB/s (6410MB/s-6410MB/s), io=10.0GiB (10.7GB), run=1675-1675msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m 1.93s user 5.48s system 76% cpu 9.655 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=897MiB/s][r=0,w=896 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=23104: Mon May 28 16:27:50 2018 write: IOPS=897, BW=898MiB/s (941MB/s)(10.0GiB/11409msec) clat (usec): min=925, max=2723, avg=1086.32, stdev=83.12 lat (usec): min=941, max=2756, avg=1112.21, stdev=84.90 clat percentiles (usec): | 1.00th=[ 955], 5.00th=[ 971], 10.00th=[ 988], 20.00th=[ 1012], | 30.00th=[ 1037], 40.00th=[ 1074], 50.00th=[ 1090], 60.00th=[ 1106], | 70.00th=[ 1123], 80.00th=[ 1139], 90.00th=[ 1172], 95.00th=[ 1221], | 99.00th=[ 1352], 99.50th=[ 1418], 99.90th=[ 1647], 99.95th=[ 1696], | 99.99th=[ 1876] bw ( KiB/s): min=903168, max=931840, per=100.00%, avg=919374.55, stdev=6756.98, samples=22 iops : min= 882, max= 910, avg=897.82, stdev= 6.59, samples=22 lat (usec) : 1000=14.95% lat (msec) : 2=85.04%, 4=0.01% cpu : usr=2.95%, sys=38.38%, ctx=25158, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=898MiB/s (941MB/s), 898MiB/s-898MiB/s (941MB/s-941MB/s), io=10.0GiB (10.7GB), run=11409-11409msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m 0.41s user 4.65s system 42% cpu 11.813 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=859MiB/s][r=0,w=859 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=7960: Mon May 28 16:26:01 2018 write: IOPS=865, BW=865MiB/s (907MB/s)(10.0GiB/11832msec) clat (usec): min=943, max=2346, avg=1126.23, stdev=105.80 lat (usec): min=961, max=2371, avg=1152.63, stdev=108.31 clat percentiles (usec): | 1.00th=[ 971], 5.00th=[ 996], 10.00th=[ 1012], 20.00th=[ 1037], | 30.00th=[ 1074], 40.00th=[ 1106], 50.00th=[ 1123], 60.00th=[ 1139], | 70.00th=[ 1156], 80.00th=[ 1188], 90.00th=[ 1237], 95.00th=[ 1303], | 99.00th=[ 1516], 99.50th=[ 1582], 99.90th=[ 1762], 99.95th=[ 1827], | 99.99th=[ 2024] bw ( KiB/s): min=870400, max=899072, per=99.99%, avg=886172.39, stdev=6405.73, samples=23 iops : min= 850, max= 878, avg=865.39, stdev= 6.25, samples=23 lat (usec) : 1000=7.23% lat (msec) : 2=92.75%, 4=0.02% cpu : usr=2.86%, sys=39.97%, ctx=22636, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=865MiB/s (907MB/s), 865MiB/s-865MiB/s (907MB/s-907MB/s), io=10.0GiB (10.7GB), run=11832-11832msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m 0.41s user 5.01s system 44% cpu 12.242 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=374MiB/s][r=0,w=5982 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=3770: Mon May 28 16:12:35 2018 write: IOPS=5829, BW=364MiB/s (382MB/s)(10.0GiB/28103msec) clat (usec): min=111, max=4706, avg=168.45, stdev=89.00 lat (usec): min=112, max=4707, avg=169.75, stdev=89.09 clat percentiles (usec): | 1.00th=[ 124], 5.00th=[ 130], 10.00th=[ 137], 20.00th=[ 145], | 30.00th=[ 151], 40.00th=[ 157], 50.00th=[ 163], 60.00th=[ 167], | 70.00th=[ 172], 80.00th=[ 178], 90.00th=[ 190], 95.00th=[ 206], | 99.00th=[ 310], 99.50th=[ 400], 99.90th=[ 1434], 99.95th=[ 2114], | 99.99th=[ 3785] bw ( KiB/s): min=349568, max=393088, per=99.96%, avg=372967.07, stdev=9398.05, samples=56 iops : min= 5462, max= 6142, avg=5827.57, stdev=146.84, samples=56 lat (usec) : 250=97.80%, 500=1.83%, 750=0.15%, 1000=0.05% lat (msec) : 2=0.11%, 4=0.05%, 10=0.01% cpu : usr=3.08%, sys=28.96%, ctx=329085, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=364MiB/s (382MB/s), 364MiB/s-364MiB/s (382MB/s-382MB/s), io=10.0GiB (10.7GB), run=28103-28103msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64 0.97s user 8.39s system 32% cpu 28.508 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][99.7%][r=0KiB/s,w=40.7MiB/s][r=0,w=10.4k IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=12732: Mon May 28 15:31:47 2018 write: IOPS=7856, BW=30.7MiB/s (32.2MB/s)(10.0GiB/333675msec) clat (usec): min=46, max=100118, avg=125.22, stdev=126.81 lat (usec): min=46, max=100118, avg=125.49, stdev=126.85 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 73], 10.00th=[ 76], 20.00th=[ 81], | 30.00th=[ 86], 40.00th=[ 91], 50.00th=[ 97], 60.00th=[ 104], | 70.00th=[ 117], 80.00th=[ 167], 90.00th=[ 223], 95.00th=[ 251], | 99.00th=[ 310], 99.50th=[ 424], 99.90th=[ 1418], 99.95th=[ 1680], | 99.99th=[ 2835] bw ( KiB/s): min=25160, max=46000, per=99.97%, avg=31415.24, stdev=1574.17, samples=667 iops : min= 6290, max=11500, avg=7853.78, stdev=393.55, samples=667 lat (usec) : 50=0.01%, 100=55.00%, 250=39.98%, 500=4.60%, 750=0.14% lat (usec) : 1000=0.06% lat (msec) : 2=0.19%, 4=0.03%, 10=0.01%, 20=0.01%, 100=0.01% lat (msec) : 250=0.01% cpu : usr=2.71%, sys=36.66%, ctx=5325439, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=30.7MiB/s (32.2MB/s), 30.7MiB/s-30.7MiB/s (32.2MB/s-32.2MB/s), io=10.0GiB (10.7GB), run=333675-333675msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k 9.42s user 123.00s system 39% cpu 5:34.11 total # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=6087KiB/s][r=0,w=12.2k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=10231: Mon May 28 15:24:11 2018 write: IOPS=8022, BW=4011KiB/s (4107kB/s)(10.0GiB/2614167msec) clat (usec): min=42, max=111174, avg=122.73, stdev=132.47 lat (usec): min=43, max=111174, avg=122.87, stdev=132.50 clat percentiles (usec): | 1.00th=[ 64], 5.00th=[ 69], 10.00th=[ 72], 20.00th=[ 76], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 91], 60.00th=[ 97], | 70.00th=[ 109], 80.00th=[ 174], 90.00th=[ 235], 95.00th=[ 260], | 99.00th=[ 318], 99.50th=[ 420], 99.90th=[ 1401], 99.95th=[ 1696], | 99.99th=[ 2868] bw ( KiB/s): min= 2946, max= 6805, per=99.99%, avg=4010.58, stdev=177.36, samples=5228 iops : min= 5892, max=13610, avg=8021.19, stdev=354.72, samples=5228 lat (usec) : 50=0.01%, 100=64.03%, 250=29.51%, 500=6.05%, 750=0.13% lat (usec) : 1000=0.05% lat (msec) : 2=0.18%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=2.66%, sys=35.36%, ctx=42570027, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=4011KiB/s (4107kB/s), 4011KiB/s-4011KiB/s (4107kB/s-4107kB/s), io=10.0GiB (10.7GB), run=2614167-2614167msec fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=51 71.79s user 928.21s system 38% cpu 43:34.81 total zpool create -o ashift=9 myzpool /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN zfs create myzpool/mydataset zfs set recordsize=8K myzpool/mydataset # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][99.2%][r=125MiB/s,w=0KiB/s][r=256k,w=0 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=14691: Mon May 28 16:57:31 2018 read: IOPS=158k, BW=77.2MiB/s (80.9MB/s)(10.0GiB/132658msec) clat (nsec): min=1460, max=445560, avg=5847.60, stdev=1345.70 lat (nsec): min=1500, max=445600, avg=5884.93, stdev=1346.19 clat percentiles (nsec): | 1.00th=[ 2008], 5.00th=[ 2768], 10.00th=[ 4832], 20.00th=[ 5216], | 30.00th=[ 5408], 40.00th=[ 5600], 50.00th=[ 5728], 60.00th=[ 5920], | 70.00th=[ 6240], 80.00th=[ 6880], 90.00th=[ 7392], 95.00th=[ 7776], | 99.00th=[ 8256], 99.50th=[ 8640], 99.90th=[15936], 99.95th=[17280], | 99.99th=[20864] bw ( KiB/s): min=70588, max=150756, per=99.80%, avg=78884.58, stdev=8458.59, samples=265 iops : min=141176, max=301512, avg=157769.43, stdev=16917.12, samples=265 lat (usec) : 2=1.00%, 4=5.56%, 10=93.24%, 20=0.20%, 50=0.01% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=8.31%, sys=91.54%, ctx=842, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=77.2MiB/s (80.9MB/s), 77.2MiB/s-77.2MiB/s (80.9MB/s-80.9MB/s), io=10.0GiB (10.7GB), run=132658-132658msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=631MiB/s,w=0KiB/s][r=162k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=8054: Mon May 28 16:58:17 2018 read: IOPS=151k, BW=589MiB/s (617MB/s)(10.0GiB/17399msec) clat (nsec): min=1730, max=1203.8k, avg=6153.04, stdev=2158.70 lat (nsec): min=1769, max=1203.0k, avg=6190.07, stdev=2160.07 clat percentiles (nsec): | 1.00th=[ 2608], 5.00th=[ 4768], 10.00th=[ 5088], 20.00th=[ 5408], | 30.00th=[ 5664], 40.00th=[ 5792], 50.00th=[ 5984], 60.00th=[ 6176], | 70.00th=[ 6496], 80.00th=[ 7008], 90.00th=[ 7648], 95.00th=[ 8032], | 99.00th=[ 8768], 99.50th=[ 9280], 99.90th=[17536], 99.95th=[35072], | 99.99th=[74240] bw ( KiB/s): min=533936, max=657600, per=99.54%, avg=599875.68, stdev=28817.13, samples=34 iops : min=133484, max=164400, avg=149969.06, stdev=7204.06, samples=34 lat (usec) : 2=0.36%, 4=3.57%, 10=95.76%, 20=0.25%, 50=0.04% lat (usec) : 100=0.02%, 250=0.01%, 500=0.01%, 750=0.01% lat (msec) : 2=0.01% cpu : usr=8.51%, sys=91.21%, ctx=557, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=589MiB/s (617MB/s), 589MiB/s-589MiB/s (617MB/s-617MB/s), io=10.0GiB (10.7GB), run=17399-17399msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][85.7%][r=1553MiB/s,w=0KiB/s][r=24.8k,w=0 IOPS][eta 00m:02s] temp: (groupid=0, jobs=1): err= 0: pid=15167: Mon May 28 16:58:48 2018 read: IOPS=12.8k, BW=800MiB/s (839MB/s)(10.0GiB/12793msec) clat (usec): min=10, max=5920.8k, avg=77.50, stdev=14652.70 lat (usec): min=11, max=5920.8k, avg=77.54, stdev=14652.70 clat percentiles (usec): | 1.00th=[ 29], 5.00th=[ 30], 10.00th=[ 31], 20.00th=[ 31], | 30.00th=[ 31], 40.00th=[ 32], 50.00th=[ 35], 60.00th=[ 39], | 70.00th=[ 41], 80.00th=[ 42], 90.00th=[ 61], 95.00th=[ 68], | 99.00th=[ 75], 99.50th=[ 80], 99.90th=[ 103], 99.95th=[ 143], | 99.99th=[ 367] bw ( MiB/s): min= 3, max= 1647, per=100.00%, avg=1391.44, stdev=458.99, samples=14 iops : min= 56, max=26356, avg=22262.93, stdev=7343.73, samples=14 lat (usec) : 20=0.05%, 50=86.96%, 100=12.87%, 250=0.11%, 500=0.01% lat (usec) : 750=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 250=0.01%, 500=0.01% cpu : usr=1.28%, sys=97.58%, ctx=366, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=800MiB/s (839MB/s), 800MiB/s-800MiB/s (839MB/s-839MB/s), io=10.0GiB (10.7GB), run=12793-12793msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=2046MiB/s,w=0KiB/s][r=2046,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=24685: Mon May 28 16:59:11 2018 read: IOPS=1980, BW=1981MiB/s (2077MB/s)(10.0GiB/5170msec) clat (usec): min=137, max=5896, avg=503.93, stdev=112.10 lat (usec): min=137, max=5896, avg=503.98, stdev=112.11 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 416], 20.00th=[ 437], | 30.00th=[ 441], 40.00th=[ 449], 50.00th=[ 474], 60.00th=[ 553], | 70.00th=[ 570], 80.00th=[ 578], 90.00th=[ 586], 95.00th=[ 594], | 99.00th=[ 619], 99.50th=[ 750], 99.90th=[ 1582], 99.95th=[ 2180], | 99.99th=[ 3392] bw ( MiB/s): min= 1936, max= 2031, per=100.00%, avg=1980.61, stdev=36.30, samples=10 iops : min= 1936, max= 2031, avg=1980.40, stdev=36.05, samples=10 lat (usec) : 250=0.55%, 500=51.47%, 750=47.49%, 1000=0.24% lat (msec) : 2=0.19%, 4=0.05%, 10=0.01% cpu : usr=0.25%, sys=99.01%, ctx=117, majf=0, minf=266 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1981MiB/s (2077MB/s), 1981MiB/s-1981MiB/s (2077MB/s-2077MB/s), io=10.0GiB (10.7GB), run=5170-5170msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [R(1)][100.0%][r=1616MiB/s,w=0KiB/s][r=1616,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=2552: Mon May 28 16:59:36 2018 read: IOPS=1658, BW=1658MiB/s (1739MB/s)(10.0GiB/6176msec) clat (usec): min=450, max=2746, avg=602.49, stdev=93.83 lat (usec): min=450, max=2746, avg=602.54, stdev=93.84 clat percentiles (usec): | 1.00th=[ 506], 5.00th=[ 515], 10.00th=[ 519], 20.00th=[ 529], | 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 553], 60.00th=[ 586], | 70.00th=[ 676], 80.00th=[ 701], 90.00th=[ 717], 95.00th=[ 742], | 99.00th=[ 791], 99.50th=[ 848], 99.90th=[ 1139], 99.95th=[ 1221], | 99.99th=[ 2343] bw ( MiB/s): min= 1604, max= 1684, per=100.00%, avg=1658.02, stdev=28.16, samples=12 iops : min= 1604, max= 1684, avg=1657.83, stdev=28.35, samples=12 lat (usec) : 500=0.05%, 750=96.24%, 1000=3.48% lat (msec) : 2=0.21%, 4=0.02% cpu : usr=0.05%, sys=99.71%, ctx=49, majf=0, minf=266 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1658MiB/s (1739MB/s), 1658MiB/s-1658MiB/s (1739MB/s-1739MB/s), io=10.0GiB (10.7GB), run=6176-6176msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=395MiB/s][r=0,w=395 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=3754: Mon May 28 17:00:04 2018 write: IOPS=400, BW=401MiB/s (420MB/s)(10.0GiB/25545msec) clat (usec): min=1643, max=27110, avg=2465.50, stdev=1415.10 lat (usec): min=1664, max=27161, avg=2492.37, stdev=1419.90 clat percentiles (usec): | 1.00th=[ 1696], 5.00th=[ 1729], 10.00th=[ 1778], 20.00th=[ 1827], | 30.00th=[ 1860], 40.00th=[ 1876], 50.00th=[ 1893], 60.00th=[ 1942], | 70.00th=[ 2212], 80.00th=[ 3163], 90.00th=[ 3916], 95.00th=[ 4621], | 99.00th=[ 7111], 99.50th=[10421], 99.90th=[19268], 99.95th=[21890], | 99.99th=[25297] bw ( KiB/s): min=329728, max=456704, per=99.92%, avg=410162.86, stdev=18017.58, samples=51 iops : min= 322, max= 446, avg=400.53, stdev=17.60, samples=51 lat (msec) : 2=63.82%, 4=27.11%, 10=8.51%, 20=0.49%, 50=0.08% cpu : usr=1.10%, sys=62.58%, ctx=108598, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=401MiB/s (420MB/s), 401MiB/s-401MiB/s (420MB/s-420MB/s), io=10.0GiB (10.7GB), run=25545-25545msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=389MiB/s][r=0,w=389 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=21583: Mon May 28 17:00:33 2018 write: IOPS=382, BW=382MiB/s (401MB/s)(10.0GiB/26781msec) clat (usec): min=1700, max=33070, avg=2584.92, stdev=1571.80 lat (usec): min=1717, max=33119, avg=2611.89, stdev=1577.07 clat percentiles (usec): | 1.00th=[ 1795], 5.00th=[ 1827], 10.00th=[ 1876], 20.00th=[ 1942], | 30.00th=[ 1975], 40.00th=[ 1991], 50.00th=[ 2024], 60.00th=[ 2057], | 70.00th=[ 2180], 80.00th=[ 3163], 90.00th=[ 4080], 95.00th=[ 4883], | 99.00th=[ 8356], 99.50th=[11207], 99.90th=[21890], 99.95th=[23987], | 99.99th=[25297] bw ( KiB/s): min=288768, max=489472, per=100.00%, avg=391556.98, stdev=23430.18, samples=53 iops : min= 282, max= 478, avg=382.36, stdev=22.88, samples=53 lat (msec) : 2=41.98%, 4=47.29%, 10=9.99%, 20=0.59%, 50=0.15% cpu : usr=1.14%, sys=63.74%, ctx=88369, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=382MiB/s (401MB/s), 382MiB/s-382MiB/s (401MB/s-401MB/s), io=10.0GiB (10.7GB), run=26781-26781msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=238MiB/s][r=0,w=3815 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=6891: Mon May 28 17:01:19 2018 write: IOPS=3696, BW=231MiB/s (242MB/s)(10.0GiB/44323msec) clat (usec): min=149, max=32142, avg=267.35, stdev=328.97 lat (usec): min=150, max=32144, avg=268.77, stdev=329.14 clat percentiles (usec): | 1.00th=[ 167], 5.00th=[ 176], 10.00th=[ 180], 20.00th=[ 190], | 30.00th=[ 202], 40.00th=[ 212], 50.00th=[ 221], 60.00th=[ 235], | 70.00th=[ 251], 80.00th=[ 285], 90.00th=[ 367], 95.00th=[ 465], | 99.00th=[ 865], 99.50th=[ 1188], 99.90th=[ 4490], 99.95th=[ 7373], | 99.99th=[13173] bw ( KiB/s): min=211712, max=292992, per=99.88%, avg=236285.72, stdev=11276.86, samples=88 iops : min= 3308, max= 4578, avg=3691.94, stdev=176.21, samples=88 lat (usec) : 250=69.06%, 500=26.95%, 750=2.58%, 1000=0.70% lat (msec) : 2=0.48%, 4=0.11%, 10=0.09%, 20=0.02%, 50=0.01% cpu : usr=1.61%, sys=46.24%, ctx=361741, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=231MiB/s (242MB/s), 231MiB/s-231MiB/s (242MB/s-242MB/s), io=10.0GiB (10.7GB), run=44323-44323msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=38.7MiB/s][r=0,w=9907 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=15271: Mon May 28 17:06:58 2018 write: IOPS=7791, BW=30.4MiB/s (31.9MB/s)(10.0GiB/336464msec) clat (usec): min=50, max=21850, avg=126.25, stdev=342.25 lat (usec): min=50, max=21851, avg=126.48, stdev=342.37 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 70], 10.00th=[ 73], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 87], 50.00th=[ 92], 60.00th=[ 98], | 70.00th=[ 103], 80.00th=[ 115], 90.00th=[ 186], 95.00th=[ 227], | 99.00th=[ 375], 99.50th=[ 881], 99.90th=[ 5866], 99.95th=[ 8717], | 99.99th=[12780] bw ( KiB/s): min=26512, max=42312, per=99.93%, avg=31141.70, stdev=2827.45, samples=672 iops : min= 6628, max=10578, avg=7785.40, stdev=706.87, samples=672 lat (usec) : 100=64.38%, 250=32.39%, 500=2.46%, 750=0.22%, 1000=0.09% lat (msec) : 2=0.18%, 4=0.13%, 10=0.12%, 20=0.03%, 50=0.01% cpu : usr=2.70%, sys=32.04%, ctx=5341713, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=30.4MiB/s (31.9MB/s), 30.4MiB/s-30.4MiB/s (31.9MB/s-31.9MB/s), io=10.0GiB (10.7GB), run=336464-336464msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=6937KiB/s][r=0,w=13.9k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=20894: Mon May 28 17:56:26 2018 write: IOPS=7075, BW=3538KiB/s (3623kB/s)(10.0GiB/2963995msec) clat (usec): min=43, max=221786, avg=139.23, stdev=411.35 lat (usec): min=43, max=221786, avg=139.38, stdev=411.39 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 67], 10.00th=[ 70], 20.00th=[ 75], | 30.00th=[ 80], 40.00th=[ 86], 50.00th=[ 92], 60.00th=[ 97], | 70.00th=[ 108], 80.00th=[ 163], 90.00th=[ 215], 95.00th=[ 251], | 99.00th=[ 498], 99.50th=[ 1287], 99.90th=[ 6587], 99.95th=[ 9896], | 99.99th=[14615] bw ( KiB/s): min= 1997, max= 7353, per=99.96%, avg=3535.54, stdev=265.67, samples=5927 iops : min= 3994, max=14706, avg=7071.12, stdev=531.35, samples=5927 lat (usec) : 50=0.01%, 100=64.37%, 250=30.53%, 500=4.11%, 750=0.29% lat (usec) : 1000=0.11% lat (msec) : 2=0.24%, 4=0.17%, 10=0.14%, 20=0.05%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=2.48%, sys=27.99%, ctx=42594744, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=3538KiB/s (3623kB/s), 3538KiB/s-3538KiB/s (3623kB/s-3623kB/s), io=10.0GiB (10.7GB), run=2963995-2963995msec zpool create -o ashift=9 myzpool /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN zfs create myzpool/mydataset zfs set recordsize=16K myzpool/mydataset # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][99.3%][r=114MiB/s,w=0KiB/s][r=234k,w=0 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=15470: Mon May 28 18:00:25 2018 read: IOPS=137k, BW=67.1MiB/s (70.4MB/s)(10.0GiB/152600msec) clat (nsec): min=1510, max=523154, avg=6786.03, stdev=1506.60 lat (nsec): min=1540, max=523244, avg=6822.84, stdev=1507.51 clat percentiles (nsec): | 1.00th=[ 2040], 5.00th=[ 3152], 10.00th=[ 5856], 20.00th=[ 6176], | 30.00th=[ 6368], 40.00th=[ 6560], 50.00th=[ 6752], 60.00th=[ 6944], | 70.00th=[ 7200], 80.00th=[ 7712], 90.00th=[ 8256], 95.00th=[ 8640], | 99.00th=[ 9280], 99.50th=[ 9664], 99.90th=[21376], 99.95th=[22912], | 99.99th=[24960] bw ( KiB/s): min=63170, max=150866, per=99.84%, avg=68607.04, stdev=7676.03, samples=305 iops : min=126341, max=301732, avg=137214.27, stdev=15352.06, samples=305 lat (usec) : 2=0.96%, 4=4.27%, 10=94.40%, 20=0.20%, 50=0.16% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=7.57%, sys=92.22%, ctx=538, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=67.1MiB/s (70.4MB/s), 67.1MiB/s-67.1MiB/s (70.4MB/s-70.4MB/s), io=10.0GiB (10.7GB), run=152600-152600msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=624MiB/s,w=0KiB/s][r=160k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=7028: Mon May 28 18:01:06 2018 read: IOPS=135k, BW=526MiB/s (551MB/s)(10.0GiB/19476msec) clat (nsec): min=1580, max=994245, avg=6942.58, stdev=1998.71 lat (nsec): min=1610, max=994295, avg=6979.64, stdev=1999.65 clat percentiles (nsec): | 1.00th=[ 2160], 5.00th=[ 3120], 10.00th=[ 5856], 20.00th=[ 6240], | 30.00th=[ 6496], 40.00th=[ 6688], 50.00th=[ 6880], 60.00th=[ 7136], | 70.00th=[ 7456], 80.00th=[ 7904], 90.00th=[ 8512], 95.00th=[ 9024], | 99.00th=[10048], 99.50th=[10560], 99.90th=[21376], 99.95th=[23168], | 99.99th=[69120] bw ( KiB/s): min=441896, max=656632, per=99.01%, avg=533041.47, stdev=39336.17, samples=38 iops : min=110474, max=164158, avg=133260.26, stdev=9834.03, samples=38 lat (usec) : 2=0.74%, 4=5.43%, 10=92.85%, 20=0.82%, 50=0.16% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% cpu : usr=7.23%, sys=92.53%, ctx=212, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=526MiB/s (551MB/s), 526MiB/s-526MiB/s (551MB/s-551MB/s), io=10.0GiB (10.7GB), run=19476-19476msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=2584MiB/s,w=0KiB/s][r=41.3k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=19613: Mon May 28 18:01:21 2018 read: IOPS=40.9k, BW=2554MiB/s (2678MB/s)(10.0GiB/4010msec) clat (usec): min=8, max=341, avg=23.95, stdev= 5.19 lat (usec): min=8, max=341, avg=23.99, stdev= 5.20 clat percentiles (usec): | 1.00th=[ 20], 5.00th=[ 21], 10.00th=[ 22], 20.00th=[ 22], | 30.00th=[ 22], 40.00th=[ 23], 50.00th=[ 23], 60.00th=[ 24], | 70.00th=[ 26], 80.00th=[ 27], 90.00th=[ 28], 95.00th=[ 28], | 99.00th=[ 42], 99.50th=[ 57], 99.90th=[ 87], 99.95th=[ 92], | 99.99th=[ 118] bw ( MiB/s): min= 2345, max= 2611, per=99.92%, avg=2551.52, stdev=85.31, samples=8 iops : min=37522, max=41782, avg=40823.88, stdev=1365.28, samples=8 lat (usec) : 10=0.44%, 20=1.14%, 50=97.71%, 100=0.68%, 250=0.03% lat (usec) : 500=0.01% cpu : usr=2.39%, sys=97.33%, ctx=59, majf=0, minf=25 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=2554MiB/s (2678MB/s), 2554MiB/s-2554MiB/s (2678MB/s-2678MB/s), io=10.0GiB (10.7GB), run=4010-4010msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=3303MiB/s,w=0KiB/s][r=3303,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=3030: Mon May 28 18:01:35 2018 read: IOPS=3231, BW=3231MiB/s (3388MB/s)(10.0GiB/3169msec) clat (usec): min=104, max=974, avg=308.71, stdev=38.60 lat (usec): min=104, max=974, avg=308.75, stdev=38.60 clat percentiles (usec): | 1.00th=[ 255], 5.00th=[ 262], 10.00th=[ 265], 20.00th=[ 269], | 30.00th=[ 285], 40.00th=[ 293], 50.00th=[ 297], 60.00th=[ 310], | 70.00th=[ 334], 80.00th=[ 347], 90.00th=[ 367], 95.00th=[ 371], | 99.00th=[ 379], 99.50th=[ 383], 99.90th=[ 469], 99.95th=[ 545], | 99.99th=[ 644] bw ( MiB/s): min= 3120, max= 3338, per=99.65%, avg=3219.93, stdev=82.19, samples=6 iops : min= 3120, max= 3338, avg=3219.83, stdev=82.18, samples=6 lat (usec) : 250=0.10%, 500=99.83%, 750=0.06%, 1000=0.01% cpu : usr=0.00%, sys=99.84%, ctx=30, majf=0, minf=265 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=3231MiB/s (3388MB/s), 3231MiB/s-3231MiB/s (3388MB/s-3388MB/s), io=10.0GiB (10.7GB), run=3169-3169msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [R(1)][100.0%][r=2761MiB/s,w=0KiB/s][r=2760,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=15594: Mon May 28 18:01:50 2018 read: IOPS=2737, BW=2738MiB/s (2871MB/s)(10.0GiB/3740msec) clat (usec): min=90, max=943, avg=364.64, stdev=48.81 lat (usec): min=90, max=944, avg=364.71, stdev=48.82 clat percentiles (usec): | 1.00th=[ 302], 5.00th=[ 306], 10.00th=[ 314], 20.00th=[ 330], | 30.00th=[ 338], 40.00th=[ 347], 50.00th=[ 351], 60.00th=[ 355], | 70.00th=[ 379], 80.00th=[ 408], 90.00th=[ 441], 95.00th=[ 457], | 99.00th=[ 498], 99.50th=[ 510], 99.90th=[ 562], 99.95th=[ 619], | 99.99th=[ 799] bw ( MiB/s): min= 2636, max= 2856, per=99.36%, avg=2720.33, stdev=89.38, samples=7 iops : min= 2636, max= 2856, avg=2720.29, stdev=89.30, samples=7 lat (usec) : 100=0.18%, 250=0.02%, 500=98.87%, 750=0.92%, 1000=0.02% cpu : usr=0.16%, sys=99.71%, ctx=9, majf=0, minf=267 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=2738MiB/s (2871MB/s), 2738MiB/s-2738MiB/s (2871MB/s-2871MB/s), io=10.0GiB (10.7GB), run=3740-3740msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=539MiB/s][r=0,w=538 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=16490: Mon May 28 18:02:11 2018 write: IOPS=528, BW=528MiB/s (554MB/s)(10.0GiB/19379msec) clat (usec): min=1256, max=102593, avg=1861.95, stdev=1109.26 lat (usec): min=1268, max=102638, avg=1890.06, stdev=1110.93 clat percentiles (usec): | 1.00th=[ 1336], 5.00th=[ 1369], 10.00th=[ 1385], 20.00th=[ 1434], | 30.00th=[ 1516], 40.00th=[ 1631], 50.00th=[ 1811], 60.00th=[ 1909], | 70.00th=[ 2024], 80.00th=[ 2180], 90.00th=[ 2409], 95.00th=[ 2638], | 99.00th=[ 3195], 99.50th=[ 3425], 99.90th=[ 3949], 99.95th=[ 4490], | 99.99th=[25035] bw ( KiB/s): min=444416, max=563200, per=99.98%, avg=540991.89, stdev=19292.32, samples=38 iops : min= 434, max= 550, avg=528.26, stdev=18.83, samples=38 lat (msec) : 2=67.90%, 4=32.01%, 10=0.07%, 50=0.01%, 250=0.01% cpu : usr=1.52%, sys=58.63%, ctx=69658, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=528MiB/s (554MB/s), 528MiB/s-528MiB/s (554MB/s-554MB/s), io=10.0GiB (10.7GB), run=19379-19379msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=511MiB/s][r=0,w=510 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=23364: Mon May 28 18:02:32 2018 write: IOPS=508, BW=509MiB/s (534MB/s)(10.0GiB/20121msec) clat (usec): min=1283, max=102739, avg=1936.09, stdev=1100.81 lat (usec): min=1302, max=102766, avg=1961.13, stdev=1102.18 clat percentiles (usec): | 1.00th=[ 1401], 5.00th=[ 1434], 10.00th=[ 1450], 20.00th=[ 1500], | 30.00th=[ 1565], 40.00th=[ 1713], 50.00th=[ 1876], 60.00th=[ 1975], | 70.00th=[ 2089], 80.00th=[ 2245], 90.00th=[ 2540], 95.00th=[ 2835], | 99.00th=[ 3458], 99.50th=[ 3687], 99.90th=[ 4293], 99.95th=[ 4817], | 99.99th=[ 7308] bw ( KiB/s): min=405504, max=557056, per=99.96%, avg=520904.32, stdev=21411.74, samples=40 iops : min= 396, max= 544, avg=508.55, stdev=20.87, samples=40 lat (msec) : 2=63.08%, 4=36.73%, 10=0.19%, 250=0.01% cpu : usr=1.49%, sys=59.72%, ctx=49724, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=509MiB/s (534MB/s), 509MiB/s-509MiB/s (534MB/s-534MB/s), io=10.0GiB (10.7GB), run=20121-20121msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=281MiB/s][r=0,w=4500 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=8017: Mon May 28 18:03:10 2018 write: IOPS=4483, BW=280MiB/s (294MB/s)(10.0GiB/36539msec) clat (usec): min=123, max=44362, avg=219.86, stdev=143.59 lat (usec): min=124, max=44363, avg=221.32, stdev=143.79 clat percentiles (usec): | 1.00th=[ 141], 5.00th=[ 147], 10.00th=[ 153], 20.00th=[ 161], | 30.00th=[ 174], 40.00th=[ 182], 50.00th=[ 194], 60.00th=[ 206], | 70.00th=[ 227], 80.00th=[ 269], 90.00th=[ 322], 95.00th=[ 367], | 99.00th=[ 498], 99.50th=[ 635], 99.90th=[ 1029], 99.95th=[ 1221], | 99.99th=[ 1942] bw ( KiB/s): min=241792, max=308992, per=99.97%, avg=286896.89, stdev=10700.93, samples=73 iops : min= 3778, max= 4828, avg=4482.74, stdev=167.21, samples=73 lat (usec) : 250=76.67%, 500=22.33%, 750=0.67%, 1000=0.22% lat (msec) : 2=0.11%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=1.95%, sys=39.65%, ctx=337815, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=280MiB/s (294MB/s), 280MiB/s-280MiB/s (294MB/s-294MB/s), io=10.0GiB (10.7GB), run=36539-36539msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][99.7%][r=0KiB/s,w=40.1MiB/s][r=0,w=10.3k IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=12969: Mon May 28 18:08:43 2018 write: IOPS=7900, BW=30.9MiB/s (32.4MB/s)(10.0GiB/331798msec) clat (usec): min=48, max=117781, avg=124.59, stdev=338.41 lat (usec): min=48, max=117782, avg=124.84, stdev=338.44 clat percentiles (usec): | 1.00th=[ 65], 5.00th=[ 70], 10.00th=[ 73], 20.00th=[ 77], | 30.00th=[ 82], 40.00th=[ 87], 50.00th=[ 93], 60.00th=[ 99], | 70.00th=[ 105], 80.00th=[ 125], 90.00th=[ 186], 95.00th=[ 223], | 99.00th=[ 351], 99.50th=[ 734], 99.90th=[ 4752], 99.95th=[ 8160], | 99.99th=[13304] bw ( KiB/s): min=25032, max=42104, per=99.94%, avg=31582.02, stdev=2390.07, samples=663 iops : min= 6258, max=10526, avg=7895.47, stdev=597.51, samples=663 lat (usec) : 50=0.01%, 100=62.62%, 250=34.54%, 500=2.14%, 750=0.20% lat (usec) : 1000=0.10% lat (msec) : 2=0.18%, 4=0.10%, 10=0.08%, 20=0.03%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=2.71%, sys=32.53%, ctx=5340791, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=30.9MiB/s (32.4MB/s), 30.9MiB/s-30.9MiB/s (32.4MB/s-32.4MB/s), io=10.0GiB (10.7GB), run=331798-331798msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=6161KiB/s][r=0,w=12.3k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=7660: Mon May 28 18:54:35 2018 write: IOPS=7626, BW=3813KiB/s (3905kB/s)(10.0GiB/2749805msec) clat (usec): min=44, max=176573, avg=129.00, stdev=373.05 lat (usec): min=44, max=176573, avg=129.16, stdev=373.09 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 68], 10.00th=[ 70], 20.00th=[ 75], | 30.00th=[ 79], 40.00th=[ 85], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 103], 80.00th=[ 147], 90.00th=[ 202], 95.00th=[ 235], | 99.00th=[ 371], 99.50th=[ 824], 99.90th=[ 5604], 99.95th=[ 9372], | 99.99th=[14091] bw ( KiB/s): min= 2383, max= 6256, per=99.96%, avg=3811.65, stdev=226.39, samples=5499 iops : min= 4766, max=12512, avg=7623.33, stdev=452.78, samples=5499 lat (usec) : 50=0.01%, 100=66.16%, 250=30.22%, 500=2.86%, 750=0.23% lat (usec) : 1000=0.10% lat (msec) : 2=0.19%, 4=0.11%, 10=0.09%, 20=0.04%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=2.64%, sys=30.15%, ctx=42578759, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=3813KiB/s (3905kB/s), 3813KiB/s-3813KiB/s (3905kB/s-3905kB/s), io=10.0GiB (10.7GB), run=2749805-2749805msec zpool create -o ashift=9 myzpool /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN zfs create myzpool/mydataset zfs set recordsize=32K myzpool/mydataset # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][99.0%][r=83.0MiB/s,w=0KiB/s][r=172k,w=0 IOPS][eta 00m:02s] temp: (groupid=0, jobs=1): err= 0: pid=27937: Mon May 28 18:59:06 2018 read: IOPS=110k, BW=53.9MiB/s (56.5MB/s)(10.0GiB/189984msec) clat (nsec): min=1390, max=1177.1k, avg=8565.02, stdev=2416.09 lat (nsec): min=1430, max=1177.2k, avg=8603.42, stdev=2417.07 clat percentiles (nsec): | 1.00th=[ 2224], 5.00th=[ 7200], 10.00th=[ 7520], 20.00th=[ 7776], | 30.00th=[ 8032], 40.00th=[ 8256], 50.00th=[ 8384], 60.00th=[ 8640], | 70.00th=[ 8896], 80.00th=[ 9280], 90.00th=[10048], 95.00th=[10816], | 99.00th=[14912], 99.50th=[16320], 99.90th=[28032], 99.95th=[29568], | 99.99th=[34048] bw ( KiB/s): min=48909, max=104778, per=99.55%, avg=54943.50, stdev=4193.74, samples=379 iops : min=97818, max=209558, avg=109887.14, stdev=8387.55, samples=379 lat (usec) : 2=0.80%, 4=2.94%, 10=85.55%, 20=10.43%, 50=0.27% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=6.10%, sys=93.60%, ctx=838, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=53.9MiB/s (56.5MB/s), 53.9MiB/s-53.9MiB/s (56.5MB/s-56.5MB/s), io=10.0GiB (10.7GB), run=189984-189984msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][96.0%][r=512MiB/s,w=0KiB/s][r=131k,w=0 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=7643: Mon May 28 18:59:48 2018 read: IOPS=109k, BW=427MiB/s (448MB/s)(10.0GiB/23959msec) clat (nsec): min=1640, max=609868, avg=8634.61, stdev=2498.79 lat (nsec): min=1690, max=610148, avg=8674.82, stdev=2500.12 clat percentiles (nsec): | 1.00th=[ 2024], 5.00th=[ 2832], 10.00th=[ 7520], 20.00th=[ 7904], | 30.00th=[ 8160], 40.00th=[ 8384], 50.00th=[ 8640], 60.00th=[ 8896], | 70.00th=[ 9280], 80.00th=[ 9664], 90.00th=[10304], 95.00th=[11200], | 99.00th=[15424], 99.50th=[16768], 99.90th=[28544], 99.95th=[30080], | 99.99th=[52992] bw ( KiB/s): min=371408, max=609772, per=98.58%, avg=431419.96, stdev=40601.05, samples=47 iops : min=92852, max=152443, avg=107854.91, stdev=10150.25, samples=47 lat (usec) : 2=0.94%, 4=6.04%, 10=79.02%, 20=13.72%, 50=0.27% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=6.27%, sys=93.45%, ctx=57, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=427MiB/s (448MB/s), 427MiB/s-427MiB/s (448MB/s-448MB/s), io=10.0GiB (10.7GB), run=23959-23959msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=3183MiB/s,w=0KiB/s][r=50.9k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=19702: Mon May 28 19:00:02 2018 read: IOPS=48.2k, BW=3014MiB/s (3160MB/s)(10.0GiB/3398msec) clat (usec): min=7, max=1249, avg=20.13, stdev= 7.86 lat (usec): min=7, max=1249, avg=20.18, stdev= 7.86 clat percentiles (usec): | 1.00th=[ 17], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 19], 40.00th=[ 19], 50.00th=[ 19], 60.00th=[ 19], | 70.00th=[ 20], 80.00th=[ 22], 90.00th=[ 25], 95.00th=[ 28], | 99.00th=[ 39], 99.50th=[ 45], 99.90th=[ 98], 99.95th=[ 113], | 99.99th=[ 289] bw ( MiB/s): min= 2220, max= 3186, per=99.05%, avg=2984.90, stdev=383.35, samples=6 iops : min=35524, max=50976, avg=47758.33, stdev=6133.67, samples=6 lat (usec) : 10=0.10%, 20=70.28%, 50=29.26%, 100=0.27%, 250=0.08% lat (usec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=3.92%, sys=94.58%, ctx=4118, majf=0, minf=25 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=3014MiB/s (3160MB/s), 3014MiB/s-3014MiB/s (3160MB/s-3160MB/s), io=10.0GiB (10.7GB), run=3398-3398msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][-.-%][r=4755MiB/s,w=0KiB/s][r=4755,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=14592: Mon May 28 19:00:14 2018 read: IOPS=4467, BW=4468MiB/s (4685MB/s)(10.0GiB/2292msec) clat (usec): min=94, max=1237, avg=223.00, stdev=56.42 lat (usec): min=94, max=1237, avg=223.05, stdev=56.44 clat percentiles (usec): | 1.00th=[ 182], 5.00th=[ 186], 10.00th=[ 188], 20.00th=[ 190], | 30.00th=[ 192], 40.00th=[ 200], 50.00th=[ 217], 60.00th=[ 221], | 70.00th=[ 225], 80.00th=[ 235], 90.00th=[ 258], 95.00th=[ 334], | 99.00th=[ 420], 99.50th=[ 457], 99.90th=[ 881], 99.95th=[ 1020], | 99.99th=[ 1172] bw ( MiB/s): min= 3366, max= 4780, per=98.42%, avg=4397.14, stdev=687.98, samples=4 iops : min= 3366, max= 4780, avg=4397.00, stdev=687.90, samples=4 lat (usec) : 100=0.03%, 250=85.32%, 500=14.35%, 750=0.14%, 1000=0.10% lat (msec) : 2=0.07% cpu : usr=0.13%, sys=98.95%, ctx=282, majf=0, minf=265 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=4468MiB/s (4685MB/s), 4468MiB/s-4468MiB/s (4685MB/s-4685MB/s), io=10.0GiB (10.7GB), run=2292-2292msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [R(1)][-.-%][r=4535MiB/s,w=0KiB/s][r=4534,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=31892: Mon May 28 19:00:26 2018 read: IOPS=4150, BW=4151MiB/s (4352MB/s)(10.0GiB/2467msec) clat (usec): min=173, max=1126, avg=240.25, stdev=56.79 lat (usec): min=173, max=1127, avg=240.33, stdev=56.81 clat percentiles (usec): | 1.00th=[ 202], 5.00th=[ 204], 10.00th=[ 204], 20.00th=[ 206], | 30.00th=[ 208], 40.00th=[ 210], 50.00th=[ 212], 60.00th=[ 221], | 70.00th=[ 247], 80.00th=[ 258], 90.00th=[ 326], 95.00th=[ 367], | 99.00th=[ 461], 99.50th=[ 486], 99.90th=[ 562], 99.95th=[ 594], | 99.99th=[ 660] bw ( MiB/s): min= 2804, max= 4544, per=97.86%, avg=4062.00, stdev=841.56, samples=4 iops : min= 2804, max= 4544, avg=4062.00, stdev=841.56, samples=4 lat (usec) : 250=74.08%, 500=25.55%, 750=0.36% lat (msec) : 2=0.01% cpu : usr=0.12%, sys=99.19%, ctx=405, majf=0, minf=267 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=4151MiB/s (4352MB/s), 4151MiB/s-4151MiB/s (4352MB/s-4352MB/s), io=10.0GiB (10.7GB), run=2467-2467msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=669MiB/s][r=0,w=669 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=9635: Mon May 28 19:00:42 2018 write: IOPS=666, BW=666MiB/s (699MB/s)(10.0GiB/15372msec) clat (usec): min=1013, max=2669, avg=1474.31, stdev=176.26 lat (usec): min=1036, max=2699, avg=1499.27, stdev=177.30 clat percentiles (usec): | 1.00th=[ 1074], 5.00th=[ 1188], 10.00th=[ 1270], 20.00th=[ 1336], | 30.00th=[ 1385], 40.00th=[ 1434], 50.00th=[ 1467], 60.00th=[ 1516], | 70.00th=[ 1549], 80.00th=[ 1598], 90.00th=[ 1647], 95.00th=[ 1762], | 99.00th=[ 2024], 99.50th=[ 2089], 99.90th=[ 2245], 99.95th=[ 2376], | 99.99th=[ 2474] bw ( KiB/s): min=655360, max=747520, per=100.00%, avg=682167.10, stdev=18006.88, samples=30 iops : min= 640, max= 730, avg=666.13, stdev=17.61, samples=30 lat (msec) : 2=98.57%, 4=1.43% cpu : usr=2.06%, sys=43.13%, ctx=32358, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=666MiB/s (699MB/s), 666MiB/s-666MiB/s (699MB/s-699MB/s), io=10.0GiB (10.7GB), run=15372-15372msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=648MiB/s][r=0,w=648 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=26815: Mon May 28 19:00:59 2018 write: IOPS=656, BW=656MiB/s (688MB/s)(10.0GiB/15600msec) clat (usec): min=1046, max=7277, avg=1494.78, stdev=197.88 lat (usec): min=1065, max=7312, avg=1520.55, stdev=199.38 clat percentiles (usec): | 1.00th=[ 1106], 5.00th=[ 1188], 10.00th=[ 1287], 20.00th=[ 1352], | 30.00th=[ 1401], 40.00th=[ 1434], 50.00th=[ 1483], 60.00th=[ 1532], | 70.00th=[ 1582], 80.00th=[ 1631], 90.00th=[ 1713], 95.00th=[ 1811], | 99.00th=[ 2089], 99.50th=[ 2180], 99.90th=[ 2343], 99.95th=[ 2409], | 99.99th=[ 3130] bw ( KiB/s): min=622592, max=700416, per=99.91%, avg=671568.61, stdev=15767.39, samples=31 iops : min= 608, max= 684, avg=655.81, stdev=15.40, samples=31 lat (msec) : 2=98.15%, 4=1.84%, 10=0.01% cpu : usr=1.88%, sys=44.76%, ctx=26546, majf=0, minf=9 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=656MiB/s (688MB/s), 656MiB/s-656MiB/s (688MB/s-688MB/s), io=10.0GiB (10.7GB), run=15600-15600msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=317MiB/s][r=0,w=5074 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=18076: Mon May 28 19:01:33 2018 write: IOPS=4907, BW=307MiB/s (322MB/s)(10.0GiB/33388msec) clat (usec): min=115, max=7366, avg=200.18, stdev=148.85 lat (usec): min=116, max=7368, avg=201.58, stdev=148.97 clat percentiles (usec): | 1.00th=[ 131], 5.00th=[ 143], 10.00th=[ 157], 20.00th=[ 165], | 30.00th=[ 172], 40.00th=[ 178], 50.00th=[ 186], 60.00th=[ 196], | 70.00th=[ 204], 80.00th=[ 210], 90.00th=[ 233], 95.00th=[ 277], | 99.00th=[ 433], 99.50th=[ 603], 99.90th=[ 2376], 99.95th=[ 3916], | 99.99th=[ 5866] bw ( KiB/s): min=282059, max=351744, per=99.94%, avg=313883.52, stdev=17947.69, samples=66 iops : min= 4407, max= 5496, avg=4904.39, stdev=280.45, samples=66 lat (usec) : 250=93.27%, 500=6.03%, 750=0.34%, 1000=0.13% lat (msec) : 2=0.12%, 4=0.06%, 10=0.05% cpu : usr=2.33%, sys=34.21%, ctx=331288, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=307MiB/s (322MB/s), 307MiB/s-307MiB/s (322MB/s-322MB/s), io=10.0GiB (10.7GB), run=33388-33388msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=41.9MiB/s][r=0,w=10.7k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=19315: Mon May 28 19:06:53 2018 write: IOPS=8210, BW=32.1MiB/s (33.6MB/s)(10.0GiB/319278msec) clat (usec): min=48, max=99761, avg=119.80, stdev=148.03 lat (usec): min=48, max=99761, avg=120.04, stdev=148.05 clat percentiles (usec): | 1.00th=[ 66], 5.00th=[ 71], 10.00th=[ 74], 20.00th=[ 79], | 30.00th=[ 83], 40.00th=[ 89], 50.00th=[ 94], 60.00th=[ 100], | 70.00th=[ 109], 80.00th=[ 145], 90.00th=[ 202], 95.00th=[ 241], | 99.00th=[ 371], 99.50th=[ 537], 99.90th=[ 1532], 99.95th=[ 1926], | 99.99th=[ 3490] bw ( KiB/s): min=26832, max=46184, per=99.95%, avg=32826.17, stdev=1490.91, samples=638 iops : min= 6708, max=11546, avg=8206.52, stdev=372.73, samples=638 lat (usec) : 50=0.01%, 100=59.81%, 250=35.99%, 500=3.63%, 750=0.27% lat (usec) : 1000=0.09% lat (msec) : 2=0.17%, 4=0.04%, 10=0.01%, 20=0.01%, 100=0.01% cpu : usr=2.80%, sys=34.75%, ctx=5347437, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=32.1MiB/s (33.6MB/s), 32.1MiB/s-32.1MiB/s (33.6MB/s-33.6MB/s), io=10.0GiB (10.7GB), run=319278-319278msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=5885KiB/s][r=0,w=11.8k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=18427: Mon May 28 19:48:54 2018 write: IOPS=8325, BW=4163KiB/s (4262kB/s)(10.0GiB/2519056msec) clat (usec): min=43, max=276967, avg=118.20, stdev=175.73 lat (usec): min=43, max=276967, avg=118.36, stdev=175.75 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 68], 10.00th=[ 71], 20.00th=[ 75], | 30.00th=[ 80], 40.00th=[ 85], 50.00th=[ 91], 60.00th=[ 96], | 70.00th=[ 104], 80.00th=[ 151], 90.00th=[ 204], 95.00th=[ 243], | 99.00th=[ 383], 99.50th=[ 545], 99.90th=[ 1549], 99.95th=[ 1958], | 99.99th=[ 3720] bw ( KiB/s): min= 2888, max= 6458, per=100.00%, avg=4161.96, stdev=183.43, samples=5038 iops : min= 5776, max=12916, avg=8323.97, stdev=366.86, samples=5038 lat (usec) : 50=0.01%, 100=65.65%, 250=29.87%, 500=3.88%, 750=0.30% lat (usec) : 1000=0.09% lat (msec) : 2=0.16%, 4=0.04%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01% cpu : usr=2.80%, sys=33.52%, ctx=42857581, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=4163KiB/s (4262kB/s), 4163KiB/s-4163KiB/s (4262kB/s-4262kB/s), io=10.0GiB (10.7GB), run=2519056-2519056msec zpool create -o ashift=9 myzpool /dev/disk/by-id/nvme-INTEL_SSDPED1D480GA_PHMB74640010480DGN zfs create myzpool/mydataset zfs set recordsize=64K myzpool/mydataset # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][99.6%][r=77.8MiB/s,w=0KiB/s][r=159k,w=0 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=12444: Mon May 28 19:54:45 2018 read: IOPS=77.0k, BW=37.6MiB/s (39.4MB/s)(10.0GiB/272293msec) clat (nsec): min=1440, max=100399k, avg=12466.43, stdev=22099.91 lat (nsec): min=1480, max=100399k, avg=12503.72, stdev=22100.03 clat percentiles (nsec): | 1.00th=[ 2352], 5.00th=[11072], 10.00th=[11456], 20.00th=[11712], | 30.00th=[11968], 40.00th=[12224], 50.00th=[12352], 60.00th=[12608], | 70.00th=[12864], 80.00th=[13248], 90.00th=[14016], 95.00th=[14912], | 99.00th=[19328], 99.50th=[21120], 99.90th=[33536], 99.95th=[35072], | 99.99th=[39168] bw ( KiB/s): min=31334, max=90764, per=99.69%, avg=38388.28, stdev=3152.10, samples=544 iops : min=62668, max=181528, avg=76776.75, stdev=6304.17, samples=544 lat (usec) : 2=0.70%, 4=2.24%, 10=0.02%, 20=96.29%, 50=0.75% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 4=0.01%, 250=0.01% cpu : usr=4.47%, sys=95.19%, ctx=471, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=37.6MiB/s (39.4MB/s), 37.6MiB/s-37.6MiB/s (39.4MB/s-39.4MB/s), io=10.0GiB (10.7GB), run=272293-272293msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][97.1%][r=403MiB/s,w=0KiB/s][r=103k,w=0 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=10679: Mon May 28 19:55:35 2018 read: IOPS=77.0k, BW=305MiB/s (319MB/s)(10.0GiB/33609msec) clat (nsec): min=1520, max=3213.1k, avg=12305.84, stdev=3848.33 lat (nsec): min=1560, max=3214.9k, avg=12343.29, stdev=3849.92 clat percentiles (nsec): | 1.00th=[ 2008], 5.00th=[ 2832], 10.00th=[11328], 20.00th=[11840], | 30.00th=[12096], 40.00th=[12352], 50.00th=[12480], 60.00th=[12736], | 70.00th=[13120], 80.00th=[13504], 90.00th=[14272], 95.00th=[15168], | 99.00th=[19584], 99.50th=[21376], 99.90th=[34048], 99.95th=[36096], | 99.99th=[41728] bw ( KiB/s): min=274626, max=600846, per=99.40%, avg=310111.76, stdev=44396.79, samples=67 iops : min=68656, max=150211, avg=77527.84, stdev=11099.17, samples=67 lat (usec) : 2=0.97%, 4=5.61%, 10=0.02%, 20=92.57%, 50=0.84% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (msec) : 4=0.01% cpu : usr=4.86%, sys=94.85%, ctx=366, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=305MiB/s (319MB/s), 305MiB/s-305MiB/s (319MB/s-319MB/s), io=10.0GiB (10.7GB), run=33609-33609msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1) temp: (groupid=0, jobs=1): err= 0: pid=5407: Mon May 28 19:55:48 2018 read: IOPS=63.8k, BW=3986MiB/s (4180MB/s)(10.0GiB/2569msec) clat (usec): min=6, max=167, avg=15.15, stdev= 2.79 lat (usec): min=6, max=167, avg=15.18, stdev= 2.80 clat percentiles (nsec): | 1.00th=[12352], 5.00th=[12864], 10.00th=[13248], 20.00th=[13760], | 30.00th=[14144], 40.00th=[14656], 50.00th=[14912], 60.00th=[15168], | 70.00th=[15552], 80.00th=[15808], 90.00th=[16512], 95.00th=[17792], | 99.00th=[22400], 99.50th=[31104], 99.90th=[43264], 99.95th=[72192], | 99.99th=[77312] bw ( MiB/s): min= 3808, max= 4243, per=99.73%, avg=3975.05, stdev=196.80, samples=5 iops : min=60936, max=67900, avg=63600.80, stdev=3148.91, samples=5 lat (usec) : 10=0.01%, 20=97.24%, 50=2.66%, 100=0.08%, 250=0.01% cpu : usr=3.74%, sys=95.99%, ctx=11, majf=0, minf=24 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=3986MiB/s (4180MB/s), 3986MiB/s-3986MiB/s (4180MB/s-4180MB/s), io=10.0GiB (10.7GB), run=2569-2569msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1) temp: (groupid=0, jobs=1): err= 0: pid=30047: Mon May 28 19:56:00 2018 read: IOPS=5614, BW=5614MiB/s (5887MB/s)(10.0GiB/1824msec) clat (usec): min=123, max=751, avg=177.28, stdev=17.46 lat (usec): min=123, max=752, avg=177.34, stdev=17.46 clat percentiles (usec): | 1.00th=[ 159], 5.00th=[ 163], 10.00th=[ 163], 20.00th=[ 165], | 30.00th=[ 167], 40.00th=[ 169], 50.00th=[ 172], 60.00th=[ 174], | 70.00th=[ 186], 80.00th=[ 192], 90.00th=[ 198], 95.00th=[ 204], | 99.00th=[ 227], 99.50th=[ 245], 99.90th=[ 293], 99.95th=[ 318], | 99.99th=[ 330] bw ( MiB/s): min= 5516, max= 5620, per=99.46%, avg=5583.59, stdev=58.59, samples=3 iops : min= 5516, max= 5620, avg=5583.33, stdev=58.39, samples=3 lat (usec) : 250=99.52%, 500=0.47%, 1000=0.01% cpu : usr=0.33%, sys=99.51%, ctx=12, majf=0, minf=265 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=5614MiB/s (5887MB/s), 5614MiB/s-5614MiB/s (5887MB/s-5887MB/s), io=10.0GiB (10.7GB), run=1824-1824msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1) temp: (groupid=0, jobs=1): err= 0: pid=21934: Mon May 28 19:56:11 2018 read: IOPS=5508, BW=5508MiB/s (5776MB/s)(10.0GiB/1859msec) clat (usec): min=138, max=1025, avg=180.97, stdev=35.97 lat (usec): min=138, max=1025, avg=181.02, stdev=35.99 clat percentiles (usec): | 1.00th=[ 155], 5.00th=[ 157], 10.00th=[ 159], 20.00th=[ 161], | 30.00th=[ 161], 40.00th=[ 163], 50.00th=[ 167], 60.00th=[ 178], | 70.00th=[ 182], 80.00th=[ 190], 90.00th=[ 210], 95.00th=[ 265], | 99.00th=[ 326], 99.50th=[ 355], 99.90th=[ 396], 99.95th=[ 416], | 99.99th=[ 553] bw ( MiB/s): min= 4402, max= 5968, per=98.09%, avg=5403.33, stdev=869.54, samples=3 iops : min= 4402, max= 5968, avg=5403.33, stdev=869.54, samples=3 lat (usec) : 250=92.76%, 500=7.22%, 750=0.01% lat (msec) : 2=0.01% cpu : usr=0.32%, sys=99.41%, ctx=15, majf=0, minf=266 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=5508MiB/s (5776MB/s), 5508MiB/s-5508MiB/s (5776MB/s-5776MB/s), io=10.0GiB (10.7GB), run=1859-1859msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=806MiB/s][r=0,w=805 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=25432: Mon May 28 19:56:24 2018 write: IOPS=797, BW=797MiB/s (836MB/s)(10.0GiB/12847msec) clat (usec): min=937, max=2315, avg=1226.38, stdev=138.49 lat (usec): min=961, max=2344, avg=1252.60, stdev=139.82 clat percentiles (usec): | 1.00th=[ 988], 5.00th=[ 1029], 10.00th=[ 1057], 20.00th=[ 1106], | 30.00th=[ 1139], 40.00th=[ 1188], 50.00th=[ 1221], 60.00th=[ 1254], | 70.00th=[ 1287], 80.00th=[ 1336], 90.00th=[ 1385], 95.00th=[ 1450], | 99.00th=[ 1663], 99.50th=[ 1745], 99.90th=[ 1844], 99.95th=[ 1893], | 99.99th=[ 2073] bw ( KiB/s): min=788950, max=851968, per=99.91%, avg=815482.32, stdev=15087.15, samples=25 iops : min= 770, max= 832, avg=796.32, stdev=14.75, samples=25 lat (usec) : 1000=2.13% lat (msec) : 2=97.83%, 4=0.04% cpu : usr=2.31%, sys=35.08%, ctx=24495, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=797MiB/s (836MB/s), 797MiB/s-797MiB/s (836MB/s-836MB/s), io=10.0GiB (10.7GB), run=12847-12847msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=782MiB/s][r=0,w=782 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=29989: Mon May 28 19:56:38 2018 write: IOPS=778, BW=778MiB/s (816MB/s)(10.0GiB/13158msec) clat (usec): min=943, max=2521, avg=1259.41, stdev=152.32 lat (usec): min=963, max=2532, avg=1282.10, stdev=154.16 clat percentiles (usec): | 1.00th=[ 996], 5.00th=[ 1045], 10.00th=[ 1074], 20.00th=[ 1123], | 30.00th=[ 1172], 40.00th=[ 1205], 50.00th=[ 1254], 60.00th=[ 1287], | 70.00th=[ 1336], 80.00th=[ 1369], 90.00th=[ 1450], 95.00th=[ 1532], | 99.00th=[ 1729], 99.50th=[ 1778], 99.90th=[ 1942], 99.95th=[ 2057], | 99.99th=[ 2212] bw ( KiB/s): min=757760, max=819200, per=99.99%, avg=796865.50, stdev=13916.97, samples=26 iops : min= 740, max= 800, avg=778.15, stdev=13.61, samples=26 lat (usec) : 1000=1.11% lat (msec) : 2=98.82%, 4=0.07% cpu : usr=2.09%, sys=36.60%, ctx=22655, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=778MiB/s (816MB/s), 778MiB/s-778MiB/s (816MB/s-816MB/s), io=10.0GiB (10.7GB), run=13158-13158msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=356MiB/s][r=0,w=5692 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=12831: Mon May 28 19:57:07 2018 write: IOPS=5762, BW=360MiB/s (378MB/s)(10.0GiB/28430msec) clat (usec): min=112, max=21890, avg=170.29, stdev=110.16 lat (usec): min=112, max=21892, avg=171.64, stdev=110.23 clat percentiles (usec): | 1.00th=[ 125], 5.00th=[ 131], 10.00th=[ 137], 20.00th=[ 147], | 30.00th=[ 153], 40.00th=[ 159], 50.00th=[ 165], 60.00th=[ 169], | 70.00th=[ 174], 80.00th=[ 180], 90.00th=[ 194], 95.00th=[ 212], | 99.00th=[ 326], 99.50th=[ 396], 99.90th=[ 1352], 99.95th=[ 2212], | 99.99th=[ 3916] bw ( KiB/s): min=353792, max=384256, per=99.93%, avg=368570.73, stdev=7538.32, samples=56 iops : min= 5528, max= 6004, avg=5758.88, stdev=117.82, samples=56 lat (usec) : 250=97.50%, 500=2.17%, 750=0.13%, 1000=0.04% lat (msec) : 2=0.11%, 4=0.05%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=3.29%, sys=28.19%, ctx=329305, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=360MiB/s (378MB/s), 360MiB/s-360MiB/s (378MB/s-378MB/s), io=10.0GiB (10.7GB), run=28430-28430msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=40.7MiB/s][r=0,w=10.4k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=16022: Mon May 28 20:03:00 2018 write: IOPS=7428, BW=29.0MiB/s (30.4MB/s)(10.0GiB/352911msec) clat (usec): min=49, max=101811, avg=132.44, stdev=134.54 lat (usec): min=49, max=101812, avg=132.69, stdev=134.57 clat percentiles (usec): | 1.00th=[ 69], 5.00th=[ 75], 10.00th=[ 78], 20.00th=[ 83], | 30.00th=[ 89], 40.00th=[ 95], 50.00th=[ 102], 60.00th=[ 110], | 70.00th=[ 135], 80.00th=[ 188], 90.00th=[ 229], 95.00th=[ 262], | 99.00th=[ 326], 99.50th=[ 420], 99.90th=[ 1385], 99.95th=[ 1582], | 99.99th=[ 2507] bw ( KiB/s): min=22704, max=44040, per=99.94%, avg=29695.30, stdev=1625.27, samples=705 iops : min= 5676, max=11010, avg=7423.80, stdev=406.32, samples=705 lat (usec) : 50=0.01%, 100=47.18%, 250=46.33%, 500=6.08%, 750=0.14% lat (usec) : 1000=0.06% lat (msec) : 2=0.19%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 250=0.01% cpu : usr=2.82%, sys=35.27%, ctx=5307723, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=29.0MiB/s (30.4MB/s), 29.0MiB/s-29.0MiB/s (30.4MB/s-30.4MB/s), io=10.0GiB (10.7GB), run=352911-352911msec # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=6043KiB/s][r=0,w=12.1k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=21638: Mon May 28 20:49:06 2018 write: IOPS=7584, BW=3792KiB/s (3883kB/s)(10.0GiB/2764905msec) clat (usec): min=45, max=170944, avg=129.75, stdev=138.83 lat (usec): min=45, max=170944, avg=129.94, stdev=138.85 clat percentiles (usec): | 1.00th=[ 67], 5.00th=[ 72], 10.00th=[ 75], 20.00th=[ 80], | 30.00th=[ 86], 40.00th=[ 92], 50.00th=[ 98], 60.00th=[ 106], | 70.00th=[ 139], 80.00th=[ 186], 90.00th=[ 225], 95.00th=[ 258], | 99.00th=[ 326], 99.50th=[ 412], 99.90th=[ 1369], 99.95th=[ 1582], | 99.99th=[ 2540] bw ( KiB/s): min= 2430, max= 6303, per=99.99%, avg=3791.64, stdev=190.65, samples=5529 iops : min= 4860, max=12606, avg=7583.29, stdev=381.30, samples=5529 lat (usec) : 50=0.01%, 100=52.72%, 250=41.41%, 500=5.48%, 750=0.13% lat (usec) : 1000=0.05% lat (msec) : 2=0.18%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=2.70%, sys=34.48%, ctx=42533942, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=3792KiB/s (3883kB/s), 3792KiB/s-3792KiB/s (3883kB/s-3883kB/s), io=10.0GiB (10.7GB), run=2764905-2764905msec BTRFS ~ # ~/bench temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][95.2%][r=520MiB/s,w=0KiB/s][r=1064k,w=0 IOPS][eta 00m:04s] temp: (groupid=0, jobs=1): err= 0: pid=24736: Tue May 29 09:46:27 2018 read: IOPS=260k, BW=127MiB/s (133MB/s)(10.0GiB/80760msec) clat (nsec): min=239, max=1171.8k, avg=3385.14, stdev=7370.64 lat (nsec): min=269, max=1171.8k, avg=3424.32, stdev=7377.47 clat percentiles (nsec): | 1.00th=[ 370], 5.00th=[ 450], 10.00th=[ 510], 20.00th=[ 540], | 30.00th=[ 588], 40.00th=[ 596], 50.00th=[ 604], 60.00th=[ 620], | 70.00th=[ 684], 80.00th=[ 852], 90.00th=[20864], 95.00th=[22144], | 99.00th=[26752], 99.50th=[27520], 99.90th=[30080], 99.95th=[37120], | 99.99th=[48384] bw ( KiB/s): min=22203, max=570323, per=98.73%, avg=128183.95, stdev=153475.59, samples=161 iops : min=44406, max=1140646, avg=256368.04, stdev=306951.27, samples=161 lat (nsec) : 250=0.01%, 500=8.09%, 750=66.62%, 1000=9.59% lat (usec) : 2=2.34%, 4=0.72%, 10=0.13%, 20=0.46%, 50=12.04% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (msec) : 2=0.01% cpu : usr=12.94%, sys=41.41%, ctx=2621887, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=127MiB/s (133MB/s), 127MiB/s-127MiB/s (133MB/s-133MB/s), io=10.0GiB (10.7GB), run=80760-80760msec temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=183MiB/s,w=0KiB/s][r=46.9k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=24758: Tue May 29 09:47:38 2018 read: IOPS=43.0k, BW=172MiB/s (180MB/s)(10.0GiB/59585msec) clat (usec): min=17, max=187, avg=21.90, stdev= 2.51 lat (usec): min=17, max=187, avg=21.95, stdev= 2.52 clat percentiles (nsec): | 1.00th=[19072], 5.00th=[19840], 10.00th=[20096], 20.00th=[20608], | 30.00th=[20864], 40.00th=[21120], 50.00th=[21376], 60.00th=[21632], | 70.00th=[21888], 80.00th=[22656], 90.00th=[24192], 95.00th=[25984], | 99.00th=[31616], 99.50th=[34048], 99.90th=[46336], 99.95th=[48896], | 99.99th=[52480] bw ( KiB/s): min=118650, max=191216, per=99.98%, avg=175942.22, stdev=12204.43, samples=119 iops : min=29662, max=47804, avg=43985.55, stdev=3051.15, samples=119 lat (usec) : 20=8.68%, 50=91.30%, 100=0.03%, 250=0.01% cpu : usr=2.53%, sys=36.87%, ctx=2621490, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=172MiB/s (180MB/s), 172MiB/s-172MiB/s (180MB/s-180MB/s), io=10.0GiB (10.7GB), run=59585-59585msec temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=744MiB/s,w=0KiB/s][r=11.9k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=24788: Tue May 29 09:48:02 2018 read: IOPS=11.9k, BW=741MiB/s (777MB/s)(10.0GiB/13817msec) clat (usec): min=75, max=717, avg=83.44, stdev= 3.92 lat (usec): min=75, max=717, avg=83.50, stdev= 3.92 clat percentiles (usec): | 1.00th=[ 79], 5.00th=[ 80], 10.00th=[ 81], 20.00th=[ 82], | 30.00th=[ 82], 40.00th=[ 83], 50.00th=[ 83], 60.00th=[ 84], | 70.00th=[ 85], 80.00th=[ 86], 90.00th=[ 87], 95.00th=[ 89], | 99.00th=[ 96], 99.50th=[ 102], 99.90th=[ 110], 99.95th=[ 112], | 99.99th=[ 116] bw ( KiB/s): min=746880, max=767488, per=99.95%, avg=758558.33, stdev=5171.79, samples=27 iops : min=11670, max=11992, avg=11852.44, stdev=80.81, samples=27 lat (usec) : 100=99.36%, 250=0.64%, 500=0.01%, 750=0.01% cpu : usr=0.28%, sys=32.69%, ctx=1382532, majf=0, minf=24 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=741MiB/s (777MB/s), 741MiB/s-741MiB/s (777MB/s-777MB/s), io=10.0GiB (10.7GB), run=13817-13817msec temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=942MiB/s,w=0KiB/s][r=942,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=24808: Tue May 29 09:48:22 2018 read: IOPS=945, BW=945MiB/s (991MB/s)(10.0GiB/10833msec) clat (usec): min=916, max=1345, avg=1056.63, stdev=32.86 lat (usec): min=916, max=1345, avg=1056.73, stdev=32.87 clat percentiles (usec): | 1.00th=[ 1004], 5.00th=[ 1012], 10.00th=[ 1020], 20.00th=[ 1029], | 30.00th=[ 1037], 40.00th=[ 1045], 50.00th=[ 1057], 60.00th=[ 1057], | 70.00th=[ 1057], 80.00th=[ 1074], 90.00th=[ 1106], 95.00th=[ 1123], | 99.00th=[ 1156], 99.50th=[ 1172], 99.90th=[ 1205], 99.95th=[ 1205], | 99.99th=[ 1221] bw ( KiB/s): min=925892, max=997376, per=99.90%, avg=966991.62, stdev=14977.97, samples=21 iops : min= 904, max= 974, avg=944.29, stdev=14.66, samples=21 lat (usec) : 1000=0.37% lat (msec) : 2=99.63% cpu : usr=0.47%, sys=31.55%, ctx=1195024, majf=0, minf=264 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=945MiB/s (991MB/s), 945MiB/s-945MiB/s (991MB/s-991MB/s), io=10.0GiB (10.7GB), run=10833-10833msec temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [R(1)][100.0%][r=1638MiB/s,w=0KiB/s][r=1637,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=24828: Tue May 29 09:48:38 2018 read: IOPS=1636, BW=1636MiB/s (1716MB/s)(10.0GiB/6259msec) clat (usec): min=45, max=3429, avg=610.37, stdev=438.91 lat (usec): min=45, max=3429, avg=610.46, stdev=438.90 clat percentiles (usec): | 1.00th=[ 54], 5.00th=[ 57], 10.00th=[ 84], 20.00th=[ 265], | 30.00th=[ 285], 40.00th=[ 306], 50.00th=[ 351], 60.00th=[ 881], | 70.00th=[ 930], 80.00th=[ 963], 90.00th=[ 1172], 95.00th=[ 1205], | 99.00th=[ 1729], 99.50th=[ 1745], 99.90th=[ 2868], 99.95th=[ 3130], | 99.99th=[ 3326] bw ( MiB/s): min= 1628, max= 1644, per=99.99%, avg=1635.89, stdev= 3.77, samples=12 iops : min= 1628, max= 1644, avg=1635.83, stdev= 3.76, samples=12 lat (usec) : 50=0.04%, 100=11.65%, 250=2.91%, 500=38.96%, 750=0.68% lat (usec) : 1000=32.25% lat (msec) : 2=13.30%, 4=0.21% cpu : usr=0.22%, sys=52.91%, ctx=712477, majf=0, minf=266 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1636MiB/s (1716MB/s), 1636MiB/s-1636MiB/s (1716MB/s-1716MB/s), io=10.0GiB (10.7GB), run=6259-6259msec temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=713MiB/s][r=0,w=712 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=24848: Tue May 29 09:48:53 2018 write: IOPS=718, BW=719MiB/s (754MB/s)(10.0GiB/14243msec) clat (usec): min=1231, max=6178, avg=1372.79, stdev=95.79 lat (usec): min=1248, max=6186, avg=1389.71, stdev=95.99 clat percentiles (usec): | 1.00th=[ 1287], 5.00th=[ 1303], 10.00th=[ 1319], 20.00th=[ 1319], | 30.00th=[ 1336], 40.00th=[ 1352], 50.00th=[ 1352], 60.00th=[ 1369], | 70.00th=[ 1385], 80.00th=[ 1385], 90.00th=[ 1418], 95.00th=[ 1450], | 99.00th=[ 1811], 99.50th=[ 1827], 99.90th=[ 1893], 99.95th=[ 1942], | 99.99th=[ 2057] bw ( KiB/s): min=643072, max=753664, per=100.00%, avg=736275.68, stdev=23083.25, samples=28 iops : min= 628, max= 736, avg=719.00, stdev=22.53, samples=28 lat (msec) : 2=99.98%, 4=0.01%, 10=0.01% cpu : usr=1.07%, sys=40.44%, ctx=31743, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=719MiB/s (754MB/s), 719MiB/s-719MiB/s (754MB/s-754MB/s), io=10.0GiB (10.7GB), run=14243-14243msec temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=705MiB/s][r=0,w=705 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=24870: Tue May 29 09:49:09 2018 write: IOPS=709, BW=710MiB/s (744MB/s)(10.0GiB/14432msec) clat (usec): min=1238, max=16131, avg=1390.81, stdev=191.67 lat (usec): min=1252, max=16147, avg=1407.81, stdev=191.69 clat percentiles (usec): | 1.00th=[ 1303], 5.00th=[ 1319], 10.00th=[ 1336], 20.00th=[ 1352], | 30.00th=[ 1369], 40.00th=[ 1369], 50.00th=[ 1385], 60.00th=[ 1385], | 70.00th=[ 1401], 80.00th=[ 1418], 90.00th=[ 1434], 95.00th=[ 1467], | 99.00th=[ 1762], 99.50th=[ 1844], 99.90th=[ 1926], 99.95th=[ 1958], | 99.99th=[12125] bw ( KiB/s): min=651264, max=737280, per=100.00%, avg=726936.00, stdev=15873.65, samples=28 iops : min= 636, max= 720, avg=709.86, stdev=15.50, samples=28 lat (msec) : 2=99.96%, 4=0.02%, 20=0.02% cpu : usr=1.35%, sys=40.33%, ctx=31971, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=710MiB/s (744MB/s), 710MiB/s-710MiB/s (744MB/s-744MB/s), io=10.0GiB (10.7GB), run=14432-14432msec temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=218MiB/s][r=0,w=3495 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=24890: Tue May 29 09:49:53 2018 write: IOPS=3745, BW=234MiB/s (245MB/s)(10.0GiB/43747msec) clat (usec): min=176, max=24203, avg=264.48, stdev=98.12 lat (usec): min=177, max=24204, avg=265.42, stdev=98.12 clat percentiles (usec): | 1.00th=[ 223], 5.00th=[ 237], 10.00th=[ 239], 20.00th=[ 243], | 30.00th=[ 245], 40.00th=[ 249], 50.00th=[ 262], 60.00th=[ 269], | 70.00th=[ 273], 80.00th=[ 285], 90.00th=[ 297], 95.00th=[ 314], | 99.00th=[ 347], 99.50th=[ 363], 99.90th=[ 400], 99.95th=[ 416], | 99.99th=[ 453] bw ( KiB/s): min=209408, max=248448, per=99.94%, avg=239547.59, stdev=9423.17, samples=87 iops : min= 3272, max= 3882, avg=3742.92, stdev=147.24, samples=87 lat (usec) : 250=42.24%, 500=57.76%, 750=0.01% lat (msec) : 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=1.40%, sys=45.27%, ctx=498123, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=234MiB/s (245MB/s), 234MiB/s-234MiB/s (245MB/s-245MB/s), io=10.0GiB (10.7GB), run=43747-43747msec temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=21.0MiB/s][r=0,w=5380 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=24912: Tue May 29 09:59:06 2018 write: IOPS=4753, BW=18.6MiB/s (19.5MB/s)(10.0GiB/551421msec) clat (usec): min=105, max=131870, avg=208.68, stdev=321.92 lat (usec): min=105, max=131870, avg=208.84, stdev=321.92 clat percentiles (usec): | 1.00th=[ 161], 5.00th=[ 169], 10.00th=[ 172], 20.00th=[ 178], | 30.00th=[ 184], 40.00th=[ 194], 50.00th=[ 204], 60.00th=[ 212], | 70.00th=[ 225], 80.00th=[ 235], 90.00th=[ 253], 95.00th=[ 269], | 99.00th=[ 297], 99.50th=[ 306], 99.90th=[ 330], 99.95th=[ 338], | 99.99th=[ 367] bw ( KiB/s): min=12512, max=22736, per=99.98%, avg=19011.47, stdev=2048.42, samples=1102 iops : min= 3128, max= 5684, avg=4752.84, stdev=512.11, samples=1102 lat (usec) : 250=88.38%, 500=11.62%, 750=0.01% lat (msec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (msec) : 250=0.01% cpu : usr=1.44%, sys=46.80%, ctx=5896774, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=18.6MiB/s (19.5MB/s), 18.6MiB/s-18.6MiB/s (19.5MB/s-19.5MB/s), io=10.0GiB (10.7GB), run=551421-551421msec temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=2671KiB/s][r=0,w=5342 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=24962: Tue May 29 11:18:26 2018 write: IOPS=4406, BW=2203KiB/s (2256kB/s)(10.0GiB/4759023msec) clat (usec): min=104, max=564451, avg=225.15, stdev=1077.15 lat (usec): min=104, max=564451, avg=225.30, stdev=1077.15 clat percentiles (usec): | 1.00th=[ 161], 5.00th=[ 172], 10.00th=[ 178], 20.00th=[ 188], | 30.00th=[ 198], 40.00th=[ 208], 50.00th=[ 217], 60.00th=[ 227], | 70.00th=[ 237], 80.00th=[ 253], 90.00th=[ 273], 95.00th=[ 289], | 99.00th=[ 326], 99.50th=[ 347], 99.90th=[ 396], 99.95th=[ 429], | 99.99th=[ 1811] bw ( KiB/s): min= 7, max= 2791, per=100.00%, avg=2203.10, stdev=300.20, samples=9518 iops : min= 14, max= 5582, avg=4406.23, stdev=600.40, samples=9518 lat (usec) : 250=78.47%, 500=21.49%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.02%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% cpu : usr=1.37%, sys=46.19%, ctx=59787548, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=2203KiB/s (2256kB/s), 2203KiB/s-2203KiB/s (2256kB/s-2256kB/s), io=10.0GiB (10.7GB), run=4759023-4759023msec ~/bench 97.51s user 2641.79s system 48% cpu 1:34:20.95 total EXT4 # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][98.5%][r=535MiB/s,w=0KiB/s][r=1096k,w=0 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=25454: Tue May 29 12:13:21 2018 read: IOPS=331k, BW=162MiB/s (170MB/s)(10.0GiB/63314msec) clat (nsec): min=239, max=1570.0k, avg=2588.20, stdev=11380.85 lat (nsec): min=269, max=1570.0k, avg=2623.95, stdev=11382.76 clat percentiles (nsec): | 1.00th=[ 382], 5.00th=[ 458], 10.00th=[ 524], 20.00th=[ 548], | 30.00th=[ 588], 40.00th=[ 604], 50.00th=[ 620], 60.00th=[ 628], | 70.00th=[ 692], 80.00th=[ 780], 90.00th=[ 14144], 95.00th=[ 15040], | 99.00th=[ 18048], 99.50th=[ 18560], 99.90th=[ 21632], 99.95th=[ 32640], | 99.99th=[436224] bw ( KiB/s): min=25646, max=555616, per=98.64%, avg=163364.74, stdev=161422.62, samples=126 iops : min=51292, max=1111234, avg=326729.56, stdev=322845.30, samples=126 lat (nsec) : 250=0.01%, 500=8.70%, 750=69.58%, 1000=7.39% lat (usec) : 2=0.97%, 4=0.81%, 10=0.04%, 20=12.32%, 50=0.17% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=13.82%, sys=32.02%, ctx=2621472, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=20971520,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=162MiB/s (170MB/s), 162MiB/s-162MiB/s (170MB/s-170MB/s), io=10.0GiB (10.7GB), run=63314-63314msec Disk stats (read/write): nvme0n1: ios=2621427/27659, merge=0/54, ticks=30843/0, in_queue=173, util=0.27% # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=261MiB/s,w=0KiB/s][r=66.7k,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=25481: Tue May 29 12:14:13 2018 read: IOPS=64.3k, BW=251MiB/s (263MB/s)(10.0GiB/40758msec) clat (usec): min=13, max=1452, avg=15.03, stdev=18.78 lat (usec): min=13, max=1453, avg=15.08, stdev=18.78 clat percentiles (usec): | 1.00th=[ 14], 5.00th=[ 14], 10.00th=[ 14], 20.00th=[ 15], | 30.00th=[ 15], 40.00th=[ 15], 50.00th=[ 15], 60.00th=[ 15], | 70.00th=[ 15], 80.00th=[ 15], 90.00th=[ 16], 95.00th=[ 16], | 99.00th=[ 19], 99.50th=[ 22], 99.90th=[ 42], 99.95th=[ 174], | 99.99th=[ 1139] bw ( KiB/s): min=249272, max=270048, per=99.95%, avg=257135.51, stdev=6250.61, samples=81 iops : min=62318, max=67512, avg=64283.85, stdev=1562.62, samples=81 lat (usec) : 20=99.27%, 50=0.67%, 100=0.01%, 250=0.01%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=0.49%, sys=20.15%, ctx=2621466, majf=0, minf=10 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=2621440,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=251MiB/s (263MB/s), 251MiB/s-251MiB/s (263MB/s-263MB/s), io=10.0GiB (10.7GB), run=40758-40758msec Disk stats (read/write): nvme0n1: ios=2611809/11631, merge=0/32, ticks=30457/0, in_queue=107, util=0.26% # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][91.7%][r=1010MiB/s,w=0KiB/s][r=16.2k,w=0 IOPS][eta 00m:01s] temp: (groupid=0, jobs=1): err= 0: pid=25503: Tue May 29 12:14:34 2018 read: IOPS=14.0k, BW=935MiB/s (981MB/s)(10.0GiB/10948msec) clat (usec): min=59, max=214, avg=66.03, stdev= 6.56 lat (usec): min=59, max=214, avg=66.08, stdev= 6.58 clat percentiles (usec): | 1.00th=[ 60], 5.00th=[ 61], 10.00th=[ 61], 20.00th=[ 61], | 30.00th=[ 62], 40.00th=[ 62], 50.00th=[ 62], 60.00th=[ 64], | 70.00th=[ 73], 80.00th=[ 74], 90.00th=[ 75], 95.00th=[ 76], | 99.00th=[ 81], 99.50th=[ 86], 99.90th=[ 96], 99.95th=[ 99], | 99.99th=[ 104] bw ( KiB/s): min=841216, max=1034368, per=99.61%, avg=954073.43, stdev=86423.09, samples=21 iops : min=13144, max=16162, avg=14907.24, stdev=1350.29, samples=21 lat (usec) : 100=99.96%, 250=0.04% cpu : usr=1.54%, sys=22.38%, ctx=163844, majf=0, minf=26 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=163840,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=935MiB/s (981MB/s), 935MiB/s-935MiB/s (981MB/s-981MB/s), io=10.0GiB (10.7GB), run=10948-10948msec Disk stats (read/write): nvme0n1: ios=161608/2, merge=0/1, ticks=8057/0, in_queue=0, util=0.00% # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randread --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randread, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [r(1)][100.0%][r=1153MiB/s,w=0KiB/s][r=1153,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=25523: Tue May 29 12:14:53 2018 read: IOPS=1116, BW=1116MiB/s (1171MB/s)(10.0GiB/9172msec) clat (usec): min=854, max=1584, avg=894.44, stdev=63.37 lat (usec): min=854, max=1584, avg=894.55, stdev=63.41 clat percentiles (usec): | 1.00th=[ 857], 5.00th=[ 857], 10.00th=[ 865], 20.00th=[ 865], | 30.00th=[ 865], 40.00th=[ 865], 50.00th=[ 865], 60.00th=[ 873], | 70.00th=[ 873], 80.00th=[ 881], 90.00th=[ 1037], 95.00th=[ 1045], | 99.00th=[ 1057], 99.50th=[ 1057], 99.90th=[ 1074], 99.95th=[ 1074], | 99.99th=[ 1090] bw ( MiB/s): min= 950, max= 1154, per=99.93%, avg=1115.63, stdev=74.26, samples=18 iops : min= 950, max= 1154, avg=1115.56, stdev=74.22, samples=18 lat (usec) : 1000=85.05% lat (msec) : 2=14.95% cpu : usr=0.13%, sys=20.03%, ctx=81926, majf=0, minf=266 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1116MiB/s (1171MB/s), 1116MiB/s-1116MiB/s (1171MB/s-1171MB/s), io=10.0GiB (10.7GB), run=9172-9172msec Disk stats (read/write): nvme0n1: ios=79685/2, merge=0/1, ticks=7030/0, in_queue=0, util=0.00% # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=read --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [R(1)][100.0%][r=1638MiB/s,w=0KiB/s][r=1637,w=0 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=25546: Tue May 29 12:15:10 2018 read: IOPS=1636, BW=1637MiB/s (1716MB/s)(10.0GiB/6257msec) clat (usec): min=511, max=1157, avg=609.84, stdev= 7.61 lat (usec): min=511, max=1158, avg=609.98, stdev= 7.60 clat percentiles (usec): | 1.00th=[ 603], 5.00th=[ 603], 10.00th=[ 603], 20.00th=[ 611], | 30.00th=[ 611], 40.00th=[ 611], 50.00th=[ 611], 60.00th=[ 611], | 70.00th=[ 611], 80.00th=[ 611], 90.00th=[ 619], 95.00th=[ 619], | 99.00th=[ 627], 99.50th=[ 627], 99.90th=[ 635], 99.95th=[ 635], | 99.99th=[ 766] bw ( MiB/s): min= 1632, max= 1638, per=99.97%, avg=1636.02, stdev= 1.83, samples=12 iops : min= 1632, max= 1638, avg=1635.83, stdev= 1.99, samples=12 lat (usec) : 750=99.98%, 1000=0.01% lat (msec) : 2=0.01% cpu : usr=0.14%, sys=38.25%, ctx=40959, majf=0, minf=266 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=10240,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): READ: bw=1637MiB/s (1716MB/s), 1637MiB/s-1637MiB/s (1716MB/s-1716MB/s), io=10.0GiB (10.7GB), run=6257-6257msec Disk stats (read/write): nvme0n1: ios=81467/2, merge=0/1, ticks=16914/0, in_queue=11497, util=98.42% # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=write --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [W(1)][100.0%][r=0KiB/s,w=811MiB/s][r=0,w=810 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=25566: Tue May 29 12:15:24 2018 write: IOPS=793, BW=793MiB/s (832MB/s)(10.0GiB/12906msec) clat (usec): min=1129, max=2646, avg=1239.92, stdev=104.52 lat (usec): min=1141, max=2653, avg=1259.43, stdev=104.75 clat percentiles (usec): | 1.00th=[ 1139], 5.00th=[ 1156], 10.00th=[ 1172], 20.00th=[ 1188], | 30.00th=[ 1188], 40.00th=[ 1205], 50.00th=[ 1205], 60.00th=[ 1221], | 70.00th=[ 1237], 80.00th=[ 1254], 90.00th=[ 1303], 95.00th=[ 1549], | 99.00th=[ 1598], 99.50th=[ 1614], 99.90th=[ 1663], 99.95th=[ 1680], | 99.99th=[ 1942] bw ( KiB/s): min=702464, max=841728, per=99.97%, avg=812186.76, stdev=43634.52, samples=25 iops : min= 686, max= 822, avg=793.12, stdev=42.60, samples=25 lat (msec) : 2=99.99%, 4=0.01% cpu : usr=1.68%, sys=36.89%, ctx=20546, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=793MiB/s (832MB/s), 793MiB/s-793MiB/s (832MB/s-832MB/s), io=10.0GiB (10.7GB), run=12906-12906msec Disk stats (read/write): nvme0n1: ios=0/102405, merge=0/14271, ticks=0/33387, in_queue=25703, util=49.11% # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=1m --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=810MiB/s][r=0,w=809 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=25586: Tue May 29 12:15:38 2018 write: IOPS=771, BW=771MiB/s (809MB/s)(10.0GiB/13280msec) clat (usec): min=1131, max=2230, avg=1276.25, stdev=136.01 lat (usec): min=1142, max=2236, avg=1295.47, stdev=136.44 clat percentiles (usec): | 1.00th=[ 1156], 5.00th=[ 1172], 10.00th=[ 1172], 20.00th=[ 1188], | 30.00th=[ 1205], 40.00th=[ 1205], 50.00th=[ 1221], 60.00th=[ 1237], | 70.00th=[ 1254], 80.00th=[ 1287], 90.00th=[ 1549], 95.00th=[ 1582], | 99.00th=[ 1631], 99.50th=[ 1647], 99.90th=[ 1696], 99.95th=[ 1778], | 99.99th=[ 1909] bw ( KiB/s): min=647168, max=831488, per=99.84%, avg=788347.85, stdev=64155.97, samples=26 iops : min= 632, max= 812, avg=769.77, stdev=62.67, samples=26 lat (msec) : 2=99.99%, 4=0.01% cpu : usr=1.51%, sys=38.61%, ctx=20485, majf=0, minf=13 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=771MiB/s (809MB/s), 771MiB/s-771MiB/s (809MB/s-809MB/s), io=10.0GiB (10.7GB), run=13280-13280msec Disk stats (read/write): nvme0n1: ios=0/102441, merge=0/14813, ticks=0/33640, in_queue=25957, util=48.10% # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=64k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=430MiB/s][r=0,w=6881 IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=25606: Tue May 29 12:16:04 2018 write: IOPS=6667, BW=417MiB/s (437MB/s)(10.0GiB/24572msec) clat (usec): min=130, max=1893, avg=148.01, stdev=15.94 lat (usec): min=131, max=1894, avg=148.85, stdev=15.97 clat percentiles (usec): | 1.00th=[ 135], 5.00th=[ 137], 10.00th=[ 137], 20.00th=[ 139], | 30.00th=[ 139], 40.00th=[ 141], 50.00th=[ 141], 60.00th=[ 143], | 70.00th=[ 147], 80.00th=[ 163], 90.00th=[ 172], 95.00th=[ 180], | 99.00th=[ 202], 99.50th=[ 210], 99.90th=[ 227], 99.95th=[ 233], | 99.99th=[ 255] bw ( KiB/s): min=339968, max=443904, per=99.98%, avg=426631.67, stdev=22780.41, samples=49 iops : min= 5312, max= 6936, avg=6666.08, stdev=355.94, samples=49 lat (usec) : 250=99.99%, 500=0.01%, 750=0.01% lat (msec) : 2=0.01% cpu : usr=1.72%, sys=27.13%, ctx=327724, majf=0, minf=11 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,163840,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=417MiB/s (437MB/s), 417MiB/s-417MiB/s (437MB/s-437MB/s), io=10.0GiB (10.7GB), run=24572-24572msec Disk stats (read/write): nvme0n1: ios=0/491587, merge=0/173524, ticks=0/14377, in_queue=0, util=0.00% # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=4k --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=108MiB/s][r=0,w=27.7k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=25628: Tue May 29 12:18:16 2018 write: IOPS=20.1k, BW=78.6MiB/s (82.4MB/s)(10.0GiB/130284msec) clat (usec): min=17, max=3194, avg=48.56, stdev=34.70 lat (usec): min=18, max=3194, avg=48.69, stdev=34.72 clat percentiles (usec): | 1.00th=[ 19], 5.00th=[ 20], 10.00th=[ 20], 20.00th=[ 21], | 30.00th=[ 22], 40.00th=[ 23], 50.00th=[ 28], 60.00th=[ 39], | 70.00th=[ 71], 80.00th=[ 83], 90.00th=[ 102], 95.00th=[ 115], | 99.00th=[ 135], 99.50th=[ 141], 99.90th=[ 157], 99.95th=[ 163], | 99.99th=[ 180] bw ( KiB/s): min=41056, max=146800, per=99.91%, avg=80407.85, stdev=30078.80, samples=260 iops : min=10264, max=36700, avg=20101.94, stdev=7519.69, samples=260 lat (usec) : 20=20.81%, 50=39.79%, 100=28.56%, 250=10.84%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01% cpu : usr=2.85%, sys=22.66%, ctx=4155975, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,2621440,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=78.6MiB/s (82.4MB/s), 78.6MiB/s-78.6MiB/s (82.4MB/s-82.4MB/s), io=10.0GiB (10.7GB), run=130284-130284msec Disk stats (read/write): nvme0n1: ios=0/5187881, merge=0/1066077, ticks=0/78483, in_queue=293, util=0.01% # fio --ioengine=sync --sync=1 --end_fsync=1 --rw=randwrite --iodepth=1 --bs=512 --size=10GB --numjobs=1 --group_reporting=1 --name=temp --filename=temp && rm temp temp: (g=0): rw=randwrite, bs=(R) 512B-512B, (W) 512B-512B, (T) 512B-512B, ioengine=sync, iodepth=1 fio-3.6 Starting 1 process temp: Laying out IO file (1 file / 10240MiB) Jobs: 1 (f=1): [w(1)][100.0%][r=0KiB/s,w=25.3MiB/s][r=0,w=51.8k IOPS][eta 00m:00s] temp: (groupid=0, jobs=1): err= 0: pid=25657: Tue May 29 12:26:36 2018 write: IOPS=42.1k, BW=20.6MiB/s (21.6MB/s)(10.0GiB/498187msec) clat (usec): min=16, max=4455, avg=23.00, stdev=15.93 lat (usec): min=16, max=4455, avg=23.06, stdev=15.93 clat percentiles (usec): | 1.00th=[ 18], 5.00th=[ 18], 10.00th=[ 18], 20.00th=[ 18], | 30.00th=[ 18], 40.00th=[ 18], 50.00th=[ 19], 60.00th=[ 19], | 70.00th=[ 19], 80.00th=[ 20], 90.00th=[ 34], 95.00th=[ 46], | 99.00th=[ 104], 99.50th=[ 113], 99.90th=[ 129], 99.95th=[ 137], | 99.99th=[ 149] bw ( KiB/s): min= 5447, max=26567, per=100.00%, avg=21047.12, stdev=6599.24, samples=996 iops : min=10894, max=53134, avg=42094.26, stdev=13198.48, samples=996 lat (usec) : 20=81.84%, 50=13.22%, 100=3.61%, 250=1.33%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01% cpu : usr=1.22%, sys=22.54%, ctx=24293873, majf=0, minf=12 IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,20971520,0,1 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=1 Run status group 0 (all jobs): WRITE: bw=20.6MiB/s (21.6MB/s), 20.6MiB/s-20.6MiB/s (21.6MB/s-21.6MB/s), io=10.0GiB (10.7GB), run=498187-498187msec Disk stats (read/write): nvme0n1: ios=1750798/23549973, merge=0/1044099, ticks=19453/310787, in_queue=990, util=0.01% ~/bench 28.46s user 238.76s system 30% cpu 14:37.15 total