<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style id="owaParaStyle"><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0cm;
margin-bottom:.0001pt;
font-size:8.5pt;
font-family:Menlo;
color:black;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0cm;
margin-bottom:.0001pt;
font-size:8.5pt;
font-family:Menlo;
color:black;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-GB">There are some hints in </span><a href="https://wiki.openstack.org/wiki/Documentation/HypervisorTuningGuide">https://wiki.openstack.org/wiki/Documentation/HypervisorTuningGuide</a><o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB">There are some tips in </span><a href="https://www.linux-kvm.org/page/Tuning_KVM">https://www.linux-kvm.org/page/Tuning_KVM</a> too but you’d need to find the corresponding OpenStack flags on the guest/images/hosts/flavors.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Overall, there are several options so it’s recommended to establish a baseline performance on a representative work load and try the various options.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Tim<o:p></o:p></p>
<p class="MsoNormal"><span lang="EN-GB"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:36.0pt"><b><span style="font-size:12.0pt;color:black">From:
</span></b><span style="font-size:12.0pt;color:black">Manuel Sopena Ballesteros <manuel.sb@garvan.org.au><br>
<b>Date: </b>Wednesday, 1 May 2019 at 06:35<br>
<b>To: </b>"openstack-discuss@lists.openstack.org" <openstack-discuss@lists.openstack.org><br>
<b>Subject: </b>how to get best io performance from my block devices<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Dear Openstack community,
<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">I would like to have a high performance distributed database running in Openstack vms. I tried attaching dedicated nvme pci devices to
the vm but the performance is not as good as I can get from bare metal.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Bare metal:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="p1" style="margin-left:36.0pt"><span class="s1">[root@zeus-54 data]# fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">fio-3.1</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">Starting 1 process</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">Jobs: 1 (f=1): [f(1)][100.0%][r=39.5MiB/s,w=39.6MiB/s][r=10.1k,w=10.1k IOPS][eta 00m:00s]</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">test: (groupid=0, jobs=1): err= 0: pid=50892: Wed May</span><span class="apple-converted-space">
</span><span class="s1">1 22:22:45 2019</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">read: IOPS=9805, BW=38.3MiB/s (40.2MB/s)(4596MiB/120001msec)</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">slat (usec): min=39, max=6678, avg=94.72, stdev=55.78</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">clat (nsec): min=450, max=18224, avg=525.83, stdev=120.10</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">lat (usec): min=39, max=6679, avg=95.36, stdev=55.79</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">clat percentiles (nsec):</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">|</span><span class="apple-converted-space"> </span><span class="s1">1.00th=[</span><span class="apple-converted-space">
</span><span class="s1">462],</span><span class="apple-converted-space"> </span>
<span class="s1">5.00th=[</span><span class="apple-converted-space"> </span><span class="s1">478], 10.00th=[</span><span class="apple-converted-space">
</span><span class="s1">482], 20.00th=[</span><span class="apple-converted-space">
</span><span class="s1">486],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 30.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">490], 40.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">494], 50.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">502], 60.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">510],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 70.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">516], 80.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">532], 90.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">596], 95.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">676],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 99.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">860], 99.50th=[ 1048], 99.90th=[ 1384], 99.95th=[ 2480],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 99.99th=[ 3728]</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">bw (</span><span class="apple-converted-space"> </span><span class="s1">KiB/s): min=</span><span class="apple-converted-space">
</span><span class="s1">720, max=40736, per=100.00%, avg=39389.00, stdev=5317.58, samples=239</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">iops</span><span class="apple-converted-space"> </span><span class="s1">: min=</span><span class="apple-converted-space">
</span><span class="s1">180, max=10184, avg=9847.23, stdev=1329.39, samples=239</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">write: IOPS=9799, BW=38.3MiB/s (40.1MB/s)(4594MiB/120001msec)</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">slat (nsec): min=2982, max=106207, avg=4220.09, stdev=980.04</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">clat (nsec): min=407, max=18130, avg=451.48, stdev=103.71</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">lat (usec): min=3, max=111, avg= 4.74, stdev= 1.03</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">clat percentiles (nsec):</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">|</span><span class="apple-converted-space"> </span><span class="s1">1.00th=[</span><span class="apple-converted-space">
</span><span class="s1">414],</span><span class="apple-converted-space"> </span>
<span class="s1">5.00th=[</span><span class="apple-converted-space"> </span><span class="s1">418], 10.00th=[</span><span class="apple-converted-space">
</span><span class="s1">422], 20.00th=[</span><span class="apple-converted-space">
</span><span class="s1">430],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 30.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">434], 40.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">434], 50.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">438], 60.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">438],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 70.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">442], 80.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">446], 90.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">462], 95.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">588],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 99.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">700], 99.50th=[</span><span class="apple-converted-space"> </span>
<span class="s1">916], 99.90th=[ 1208], 99.95th=[ 1288],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 99.99th=[ 3536]</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">bw (</span><span class="apple-converted-space"> </span><span class="s1">KiB/s): min=</span><span class="apple-converted-space">
</span><span class="s1">752, max=42608, per=100.00%, avg=39366.63, stdev=5355.73, samples=239</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">iops</span><span class="apple-converted-space"> </span><span class="s1">: min=</span><span class="apple-converted-space">
</span><span class="s1">188, max=10652, avg=9841.64, stdev=1338.93, samples=239</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">lat (nsec) </span><span class="apple-converted-space"> </span><span class="s1">: 500=69.98%, 750=28.64%, 1000=0.90%</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">lat (usec) </span><span class="apple-converted-space"> </span><span class="s1">: 2=0.42%, 4=0.04%, 10=0.01%, 20=0.01%</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">cpu</span><span class="apple-converted-space"> </span><span class="s1">: usr=2.20%, sys=10.85%, ctx=1176675, majf=0, minf=1372</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">IO depths</span><span class="apple-converted-space"> </span><span class="s1">: 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">submit</span><span class="apple-converted-space"> </span>
<span class="s1">: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">complete</span><span class="apple-converted-space"> </span>
<span class="s1">: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">issued rwt: total=1176625,1175958,0, short=0,0,0, dropped=0,0,0</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">latency </span><span class="apple-converted-space"> </span>
<span class="s1">: target=0, window=0, percentile=100.00%, depth=1</span><o:p></o:p></p>
<p class="p2" style="margin-left:36.0pt"><o:p> </o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">Run status group 0 (all jobs):</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">READ: bw=38.3MiB/s (40.2MB/s), 38.3MiB/s-38.3MiB/s (40.2MB/s-40.2MB/s), io=4596MiB (4819MB), run=120001-120001msec</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">WRITE: bw=38.3MiB/s (40.1MB/s), 38.3MiB/s-38.3MiB/s (40.1MB/s-40.1MB/s), io=4594MiB (4817MB), run=120001-120001msec</span><o:p></o:p></p>
<p class="p2" style="margin-left:36.0pt"><o:p> </o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">Disk stats (read/write):</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">nvme9n1: ios=1174695/883620, merge=0/0, ticks=105502/72225, in_queue=192101, util=99.28%</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><o:p> </o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">From vm:</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">[centos@kudu-1 nvme0]$ sudo fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">fio-3.1</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">Starting 1 process</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">Jobs: 1 (f=1): [m(1)][100.0%][r=29.2MiB/s,w=29.7MiB/s][r=7487,w=7595 IOPS][eta 00m:00s]</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">test: (groupid=0, jobs=1): err= 0: pid=44383: Wed May</span><span class="apple-converted-space">
</span><span class="s1">1 12:22:24 2019</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">read: IOPS=6994, BW=27.3MiB/s (28.6MB/s)(3278MiB/120000msec)</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">slat (usec): min=54, max=20476, avg=115.27, stdev=71.45</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">clat (nsec): min=1757, max=31476, avg=2163.02, stdev=688.66</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">lat (usec): min=56, max=20481, avg=118.51, stdev=71.66</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">clat percentiles (nsec):</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">|</span><span class="apple-converted-space"> </span><span class="s1">1.00th=[ 1800],</span><span class="apple-converted-space">
</span><span class="s1">5.00th=[ 1832], 10.00th=[ 1864], 20.00th=[ 1992],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 30.00th=[ 2040], 40.00th=[ 2064], 50.00th=[ 2064], 60.00th=[ 2096],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 70.00th=[ 2096], 80.00th=[ 2128], 90.00th=[ 2480], 95.00th=[ 2544],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 99.00th=[ 4448], 99.50th=[ 5536], 99.90th=[11072], 99.95th=[12736],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 99.99th=[18560]</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">bw (</span><span class="apple-converted-space"> </span><span class="s1">KiB/s): min=</span><span class="apple-converted-space">
</span><span class="s1">952, max=31224, per=100.00%, avg=28153.51, stdev=4126.89, samples=237</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">iops</span><span class="apple-converted-space"> </span><span class="s1">: min=</span><span class="apple-converted-space">
</span><span class="s1">238, max= 7806, avg=7038.23, stdev=1031.70, samples=237</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">write: IOPS=6985, BW=27.3MiB/s (28.6MB/s)(3274MiB/120000msec)</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">slat (usec): min=7, max=963, avg=12.60, stdev= 6.24</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">clat (nsec): min=1662, max=199250, avg=2030.26, stdev=712.33</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">lat (usec): min=10, max=970, avg=15.68, stdev= 6.48</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">clat percentiles (nsec):</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">|</span><span class="apple-converted-space"> </span><span class="s1">1.00th=[ 1688],</span><span class="apple-converted-space">
</span><span class="s1">5.00th=[ 1720], 10.00th=[ 1736], 20.00th=[ 1864],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 30.00th=[ 1928], 40.00th=[ 1944], 50.00th=[ 1944], 60.00th=[ 1960],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 70.00th=[ 1960], 80.00th=[ 1992], 90.00th=[ 2352], 95.00th=[ 2384],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 99.00th=[ 4048], 99.50th=[ 4768], 99.90th=[11456], 99.95th=[13120],</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">| 99.99th=[19072]</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">bw (</span><span class="apple-converted-space"> </span><span class="s1">KiB/s): min=</span><span class="apple-converted-space">
</span><span class="s1">912, max=31880, per=100.00%, avg=28119.64, stdev=4176.38, samples=237</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">iops</span><span class="apple-converted-space"> </span><span class="s1">: min=</span><span class="apple-converted-space">
</span><span class="s1">228, max= 7970, avg=7029.75, stdev=1044.07, samples=237</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">lat (usec) </span><span class="apple-converted-space"> </span><span class="s1">: 2=51.56%, 4=47.17%, 10=1.03%, 20=0.22%, 50=0.01%</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">lat (usec) </span><span class="apple-converted-space"> </span><span class="s1">: 250=0.01%</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">cpu</span><span class="apple-converted-space"> </span><span class="s1">: usr=4.96%, sys=28.37%, ctx=839307, majf=0, minf=26</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">IO depths</span><span class="apple-converted-space"> </span><span class="s1">: 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">submit</span><span class="apple-converted-space"> </span>
<span class="s1">: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">complete</span><span class="apple-converted-space"> </span>
<span class="s1">: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">issued rwt: total=839283,838268,0, short=0,0,0, dropped=0,0,0</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space">
</span><span class="s1">latency </span><span class="apple-converted-space"> </span>
<span class="s1">: target=0, window=0, percentile=100.00%, depth=1</span><o:p></o:p></p>
<p class="p2" style="margin-left:36.0pt"><o:p> </o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">Run status group 0 (all jobs):</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">READ: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3278MiB (3438MB), run=120000-120000msec</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span>
<span class="s1">WRITE: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3274MiB (3434MB), run=120000-120000msec</span><o:p></o:p></p>
<p class="p2" style="margin-left:36.0pt"><o:p> </o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="s1">Disk stats (read/write):</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><span class="apple-converted-space"> </span><span class="s1">nvme0n1: ios=838322/651596, merge=0/0, ticks=83804/22119, in_queue=104773, util=70.18%</span><o:p></o:p></p>
<p class="p1" style="margin-left:36.0pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif;color:black">Is there a way I can get near bare metal performance from my nvme block devices?<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:8.0pt;font-family:"Arial",sans-serif">NOTICE<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:36.0pt"><span style="font-size:8.0pt;font-family:"Arial",sans-serif">Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally
privileged/confidential/copyright information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete
both messages. We accept no liability for the distribution of viruses or similar in electronic communications. This notice should not be removed.
<o:p></o:p></span></p>
</div>
</div>
</body>
</html>