<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style id="owaParaStyle"><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
margin-bottom:.0001pt;
font-size:8.5pt;
font-family:"Menlo","serif";
color:black;}
p.p2, li.p2, div.p2
{mso-style-name:p2;
margin:0in;
margin-bottom:.0001pt;
font-size:8.5pt;
font-family:"Menlo","serif";
color:black;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Dear Openstack community,
<o:p></o:p></span></a></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I am configuring a high performance storage vms, I decided to go to the easy path (pci-passthrough), I can spin up vms and see the pci devices, however performance
is below native/bare metal.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Native/Bare metal performance:<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">[root@zeus-54 data]# fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">fio-3.1<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">Starting 1 process<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">Jobs: 1 (f=1): [f(1)][100.0%][r=39.5MiB/s,w=39.6MiB/s][r=10.1k,w=10.1k IOPS][eta 00m:00s]<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">test: (groupid=0, jobs=1): err= 0: pid=50892: Wed May 1 22:22:45 2019<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> read: IOPS=9805, BW=38.3MiB/s (40.2MB/s)(4596MiB/120001msec)<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> slat (usec): min=39, max=6678, avg=94.72, stdev=55.78<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> clat (nsec): min=450, max=18224, avg=525.83, stdev=120.10<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> lat (usec): min=39, max=6679, avg=95.36, stdev=55.79<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> clat percentiles (nsec):<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 1.00th=[ 462], 5.00th=[ 478], 10.00th=[ 482], 20.00th=[ 486],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 30.00th=[ 490], 40.00th=[ 494], 50.00th=[ 502], 60.00th=[ 510],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 70.00th=[ 516], 80.00th=[ 532], 90.00th=[ 596], 95.00th=[ 676],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 99.00th=[ 860], 99.50th=[ 1048], 99.90th=[ 1384], 99.95th=[ 2480],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 99.99th=[ 3728]<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> bw ( KiB/s): min= 720, max=40736, per=100.00%, avg=39389.00, stdev=5317.58, samples=239<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> iops : min= 180, max=10184, avg=9847.23, stdev=1329.39, samples=239<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> write: IOPS=9799, BW=38.3MiB/s (40.1MB/s)(4594MiB/120001msec)<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> slat (nsec): min=2982, max=106207, avg=4220.09, stdev=980.04<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> clat (nsec): min=407, max=18130, avg=451.48, stdev=103.71<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> lat (usec): min=3, max=111, avg= 4.74, stdev= 1.03<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> clat percentiles (nsec):<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 1.00th=[ 414], 5.00th=[ 418], 10.00th=[ 422], 20.00th=[ 430],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 30.00th=[ 434], 40.00th=[ 434], 50.00th=[ 438], 60.00th=[ 438],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 70.00th=[ 442], 80.00th=[ 446], 90.00th=[ 462], 95.00th=[ 588],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 99.00th=[ 700], 99.50th=[ 916], 99.90th=[ 1208], 99.95th=[ 1288],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 99.99th=[ 3536]<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> bw ( KiB/s): min= 752, max=42608, per=100.00%, avg=39366.63, stdev=5355.73, samples=239<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> iops : min= 188, max=10652, avg=9841.64, stdev=1338.93, samples=239<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> lat (nsec) : 500=69.98%, 750=28.64%, 1000=0.90%<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> lat (usec) : 2=0.42%, 4=0.04%, 10=0.01%, 20=0.01%<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> cpu : usr=2.20%, sys=10.85%, ctx=1176675, majf=0, minf=1372<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> issued rwt: total=1176625,1175958,0, short=0,0,0, dropped=0,0,0<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> latency : target=0, window=0, percentile=100.00%, depth=1<o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"><o:p> </o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">Run status group 0 (all jobs):<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> READ: bw=38.3MiB/s (40.2MB/s), 38.3MiB/s-38.3MiB/s (40.2MB/s-40.2MB/s), io=4596MiB (4819MB), run=120001-120001msec<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> WRITE: bw=38.3MiB/s (40.1MB/s), 38.3MiB/s-38.3MiB/s (40.1MB/s-40.1MB/s), io=4594MiB (4817MB), run=120001-120001msec<o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"><o:p> </o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">Disk stats (read/write):<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> nvme9n1: ios=1174695/883620, merge=0/0, ticks=105502/72225, in_queue=192101, util=99.28%<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="p1"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">VM performance:<o:p></o:p></span></b></p>
<p class="p1"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></b></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">[centos@kudu-1 nvme0]$ sudo fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">fio-3.1<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">Starting 1 process<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">Jobs: 1 (f=1): [m(1)][100.0%][r=29.2MiB/s,w=29.7MiB/s][r=7487,w=7595 IOPS][eta 00m:00s]<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">test: (groupid=0, jobs=1): err= 0: pid=44383: Wed May 1 12:22:24 2019<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> read: IOPS=6994, BW=27.3MiB/s (28.6MB/s)(3278MiB/120000msec)<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> slat (usec): min=54, max=20476, avg=115.27, stdev=71.45<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> clat (nsec): min=1757, max=31476, avg=2163.02, stdev=688.66<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> lat (usec): min=56, max=20481, avg=118.51, stdev=71.66<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> clat percentiles (nsec):<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 1.00th=[ 1800], 5.00th=[ 1832], 10.00th=[ 1864], 20.00th=[ 1992],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 30.00th=[ 2040], 40.00th=[ 2064], 50.00th=[ 2064], 60.00th=[ 2096],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 70.00th=[ 2096], 80.00th=[ 2128], 90.00th=[ 2480], 95.00th=[ 2544],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 99.00th=[ 4448], 99.50th=[ 5536], 99.90th=[11072], 99.95th=[12736],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 99.99th=[18560]<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> bw ( KiB/s): min= 952, max=31224, per=100.00%, avg=28153.51, stdev=4126.89, samples=237<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> iops : min= 238, max= 7806, avg=7038.23, stdev=1031.70, samples=237<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> write: IOPS=6985, BW=27.3MiB/s (28.6MB/s)(3274MiB/120000msec)<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> slat (usec): min=7, max=963, avg=12.60, stdev= 6.24<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> clat (nsec): min=1662, max=199250, avg=2030.26, stdev=712.33<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> lat (usec): min=10, max=970, avg=15.68, stdev= 6.48<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> clat percentiles (nsec):<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 1.00th=[ 1688], 5.00th=[ 1720], 10.00th=[ 1736], 20.00th=[ 1864],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 30.00th=[ 1928], 40.00th=[ 1944], 50.00th=[ 1944], 60.00th=[ 1960],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 70.00th=[ 1960], 80.00th=[ 1992], 90.00th=[ 2352], 95.00th=[ 2384],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 99.00th=[ 4048], 99.50th=[ 4768], 99.90th=[11456], 99.95th=[13120],<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> | 99.99th=[19072]<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> bw ( KiB/s): min= 912, max=31880, per=100.00%, avg=28119.64, stdev=4176.38, samples=237<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> iops : min= 228, max= 7970, avg=7029.75, stdev=1044.07, samples=237<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> lat (usec) : 2=51.56%, 4=47.17%, 10=1.03%, 20=0.22%, 50=0.01%<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> lat (usec) : 250=0.01%<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> cpu : usr=4.96%, sys=28.37%, ctx=839307, majf=0, minf=26<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> IO depths : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> issued rwt: total=839283,838268,0, short=0,0,0, dropped=0,0,0<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> latency : target=0, window=0, percentile=100.00%, depth=1<o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"><o:p> </o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">Run status group 0 (all jobs):<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> READ: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3278MiB (3438MB), run=120000-120000msec<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> WRITE: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3274MiB (3434MB), run=120000-120000msec<o:p></o:p></span></p>
<p class="p2"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"><o:p> </o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">Disk stats (read/write):<o:p></o:p></span></p>
<p class="p1"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> nvme0n1: ios=838322/651596, merge=0/0, ticks=83804/22119, in_queue=104773, util=70.18%<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">This is my Openstack rocky configuration:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">nova.conf on controller node<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">[pci]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">alias = { "vendor_id":"10de", "product_id":"1db1", "device_type":"type-PCI", "name":"nv_v100" }<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">alias = { "vendor_id":"8086", "product_id":"0953", "device_type":"type-PCI", "name":"nvme"}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nova.conf on compute node:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">[pci]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">passthrough_whitelist = [ {"address":"0000:84:00.0"}, {"address":"0000:85:00.0"}, {"address":"0000:86:00.0"}, {"address":"0000:87:00.0"} ]<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">alias = { "vendor_id":"8086", "product_id":"0953", "device_type":"type-PCI", "name":"nvme"}<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D">This is how the nvmes are exposed to the vm<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> <hostdev mode='subsystem' type='pci' managed='yes'><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> <driver name='vfio'/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> <source><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> <address domain='0x0000' bus='0x86' slot='0x00' function='0x0'/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> </source><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> <alias name='hostdev1'/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:Consolas;color:#1F497D"> </hostdev><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Guest OS is centos 7.6 so I am guessing nvme drivers are included.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Any help about what needs to my configuration to get close to native io performance?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thank you very much<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Manuel<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Manuel Sopena Ballesteros [mailto:manuel.sb@garvan.org.au]
<br>
<b>Sent:</b> Wednesday, May 1, 2019 10:31 PM<br>
<b>To:</b> openstack-discuss@lists.openstack.org<br>
<b>Subject:</b> how to get best io performance from my block devices<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Dear Openstack community,
<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">I would like to have a high performance distributed database running in Openstack vms. I tried attaching dedicated nvme pci devices to the vm but the performance
is not as good as I can get from bare metal.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Bare metal:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="p1"><span class="s1">[root@zeus-54 data]# fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based</span><o:p></o:p></p>
<p class="p1"><span class="s1">test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1</span><o:p></o:p></p>
<p class="p1"><span class="s1">fio-3.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">Starting 1 process</span><o:p></o:p></p>
<p class="p1"><span class="s1">Jobs: 1 (f=1): [f(1)][100.0%][r=39.5MiB/s,w=39.6MiB/s][r=10.1k,w=10.1k IOPS][eta 00m:00s]</span><o:p></o:p></p>
<p class="p1"><span class="s1">test: (groupid=0, jobs=1): err= 0: pid=50892: Wed May</span><span class="apple-converted-space">
</span><span class="s1">1 22:22:45 2019</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">read: IOPS=9805, BW=38.3MiB/s (40.2MB/s)(4596MiB/120001msec)</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">slat (usec): min=39, max=6678, avg=94.72, stdev=55.78</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">clat (nsec): min=450, max=18224, avg=525.83, stdev=120.10</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">lat (usec): min=39, max=6679, avg=95.36, stdev=55.79</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">clat percentiles (nsec):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">|</span><span class="apple-converted-space">
</span><span class="s1">1.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">462],</span><span class="apple-converted-space"> </span><span class="s1">5.00th=[</span><span class="apple-converted-space">
</span><span class="s1">478], 10.00th=[</span><span class="apple-converted-space">
</span><span class="s1">482], 20.00th=[</span><span class="apple-converted-space">
</span><span class="s1">486],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 30.00th=[</span><span class="apple-converted-space">
</span><span class="s1">490], 40.00th=[</span><span class="apple-converted-space">
</span><span class="s1">494], 50.00th=[</span><span class="apple-converted-space">
</span><span class="s1">502], 60.00th=[</span><span class="apple-converted-space">
</span><span class="s1">510],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 70.00th=[</span><span class="apple-converted-space">
</span><span class="s1">516], 80.00th=[</span><span class="apple-converted-space">
</span><span class="s1">532], 90.00th=[</span><span class="apple-converted-space">
</span><span class="s1">596], 95.00th=[</span><span class="apple-converted-space">
</span><span class="s1">676],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 99.00th=[</span><span class="apple-converted-space">
</span><span class="s1">860], 99.50th=[ 1048], 99.90th=[ 1384], 99.95th=[ 2480],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 99.99th=[ 3728]</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">bw (</span><span class="apple-converted-space">
</span><span class="s1">KiB/s): min=</span><span class="apple-converted-space"> </span>
<span class="s1">720, max=40736, per=100.00%, avg=39389.00, stdev=5317.58, samples=239</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">iops</span><span class="apple-converted-space">
</span><span class="s1">: min=</span><span class="apple-converted-space"> </span>
<span class="s1">180, max=10184, avg=9847.23, stdev=1329.39, samples=239</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">write: IOPS=9799, BW=38.3MiB/s (40.1MB/s)(4594MiB/120001msec)</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">slat (nsec): min=2982, max=106207, avg=4220.09, stdev=980.04</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">clat (nsec): min=407, max=18130, avg=451.48, stdev=103.71</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">lat (usec): min=3, max=111, avg= 4.74, stdev= 1.03</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">clat percentiles (nsec):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">|</span><span class="apple-converted-space">
</span><span class="s1">1.00th=[</span><span class="apple-converted-space"> </span>
<span class="s1">414],</span><span class="apple-converted-space"> </span><span class="s1">5.00th=[</span><span class="apple-converted-space">
</span><span class="s1">418], 10.00th=[</span><span class="apple-converted-space">
</span><span class="s1">422], 20.00th=[</span><span class="apple-converted-space">
</span><span class="s1">430],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 30.00th=[</span><span class="apple-converted-space">
</span><span class="s1">434], 40.00th=[</span><span class="apple-converted-space">
</span><span class="s1">434], 50.00th=[</span><span class="apple-converted-space">
</span><span class="s1">438], 60.00th=[</span><span class="apple-converted-space">
</span><span class="s1">438],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 70.00th=[</span><span class="apple-converted-space">
</span><span class="s1">442], 80.00th=[</span><span class="apple-converted-space">
</span><span class="s1">446], 90.00th=[</span><span class="apple-converted-space">
</span><span class="s1">462], 95.00th=[</span><span class="apple-converted-space">
</span><span class="s1">588],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 99.00th=[</span><span class="apple-converted-space">
</span><span class="s1">700], 99.50th=[</span><span class="apple-converted-space">
</span><span class="s1">916], 99.90th=[ 1208], 99.95th=[ 1288],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 99.99th=[ 3536]</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">bw (</span><span class="apple-converted-space">
</span><span class="s1">KiB/s): min=</span><span class="apple-converted-space"> </span>
<span class="s1">752, max=42608, per=100.00%, avg=39366.63, stdev=5355.73, samples=239</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">iops</span><span class="apple-converted-space">
</span><span class="s1">: min=</span><span class="apple-converted-space"> </span>
<span class="s1">188, max=10652, avg=9841.64, stdev=1338.93, samples=239</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">lat (nsec)
</span><span class="apple-converted-space"> </span><span class="s1">: 500=69.98%, 750=28.64%, 1000=0.90%</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">lat (usec)
</span><span class="apple-converted-space"> </span><span class="s1">: 2=0.42%, 4=0.04%, 10=0.01%, 20=0.01%</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cpu</span><span class="apple-converted-space">
</span><span class="s1">: usr=2.20%, sys=10.85%, ctx=1176675, majf=0, minf=1372</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">IO depths</span><span class="apple-converted-space">
</span><span class="s1">: 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">submit</span><span class="apple-converted-space">
</span><span class="s1">: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">complete</span><span class="apple-converted-space">
</span><span class="s1">: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">issued rwt: total=1176625,1175958,0, short=0,0,0, dropped=0,0,0</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">latency
</span><span class="apple-converted-space"> </span><span class="s1">: target=0, window=0, percentile=100.00%, depth=1</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">Run status group 0 (all jobs):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">READ: bw=38.3MiB/s (40.2MB/s), 38.3MiB/s-38.3MiB/s (40.2MB/s-40.2MB/s), io=4596MiB (4819MB), run=120001-120001msec</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">WRITE: bw=38.3MiB/s (40.1MB/s), 38.3MiB/s-38.3MiB/s (40.1MB/s-40.1MB/s), io=4594MiB (4817MB), run=120001-120001msec</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">Disk stats (read/write):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">nvme9n1: ios=1174695/883620, merge=0/0, ticks=105502/72225, in_queue=192101, util=99.28%</span><o:p></o:p></p>
<p class="p1"><o:p> </o:p></p>
<p class="p1"><span class="s1">From vm:</span><o:p></o:p></p>
<p class="p1"><span class="s1">[centos@kudu-1 nvme0]$ sudo fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based</span><o:p></o:p></p>
<p class="p1"><span class="s1">test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1</span><o:p></o:p></p>
<p class="p1"><span class="s1">fio-3.1</span><o:p></o:p></p>
<p class="p1"><span class="s1">Starting 1 process</span><o:p></o:p></p>
<p class="p1"><span class="s1">Jobs: 1 (f=1): [m(1)][100.0%][r=29.2MiB/s,w=29.7MiB/s][r=7487,w=7595 IOPS][eta 00m:00s]</span><o:p></o:p></p>
<p class="p1"><span class="s1">test: (groupid=0, jobs=1): err= 0: pid=44383: Wed May</span><span class="apple-converted-space">
</span><span class="s1">1 12:22:24 2019</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">read: IOPS=6994, BW=27.3MiB/s (28.6MB/s)(3278MiB/120000msec)</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">slat (usec): min=54, max=20476, avg=115.27, stdev=71.45</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">clat (nsec): min=1757, max=31476, avg=2163.02, stdev=688.66</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">lat (usec): min=56, max=20481, avg=118.51, stdev=71.66</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">clat percentiles (nsec):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">|</span><span class="apple-converted-space">
</span><span class="s1">1.00th=[ 1800],</span><span class="apple-converted-space">
</span><span class="s1">5.00th=[ 1832], 10.00th=[ 1864], 20.00th=[ 1992],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 30.00th=[ 2040], 40.00th=[ 2064], 50.00th=[ 2064], 60.00th=[ 2096],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 70.00th=[ 2096], 80.00th=[ 2128], 90.00th=[ 2480], 95.00th=[ 2544],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 99.00th=[ 4448], 99.50th=[ 5536], 99.90th=[11072], 99.95th=[12736],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 99.99th=[18560]</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">bw (</span><span class="apple-converted-space">
</span><span class="s1">KiB/s): min=</span><span class="apple-converted-space"> </span>
<span class="s1">952, max=31224, per=100.00%, avg=28153.51, stdev=4126.89, samples=237</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">iops</span><span class="apple-converted-space">
</span><span class="s1">: min=</span><span class="apple-converted-space"> </span>
<span class="s1">238, max= 7806, avg=7038.23, stdev=1031.70, samples=237</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">write: IOPS=6985, BW=27.3MiB/s (28.6MB/s)(3274MiB/120000msec)</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">slat (usec): min=7, max=963, avg=12.60, stdev= 6.24</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">clat (nsec): min=1662, max=199250, avg=2030.26, stdev=712.33</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">lat (usec): min=10, max=970, avg=15.68, stdev= 6.48</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">clat percentiles (nsec):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">|</span><span class="apple-converted-space">
</span><span class="s1">1.00th=[ 1688],</span><span class="apple-converted-space">
</span><span class="s1">5.00th=[ 1720], 10.00th=[ 1736], 20.00th=[ 1864],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 30.00th=[ 1928], 40.00th=[ 1944], 50.00th=[ 1944], 60.00th=[ 1960],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 70.00th=[ 1960], 80.00th=[ 1992], 90.00th=[ 2352], 95.00th=[ 2384],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 99.00th=[ 4048], 99.50th=[ 4768], 99.90th=[11456], 99.95th=[13120],</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">| 99.99th=[19072]</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">bw (</span><span class="apple-converted-space">
</span><span class="s1">KiB/s): min=</span><span class="apple-converted-space"> </span>
<span class="s1">912, max=31880, per=100.00%, avg=28119.64, stdev=4176.38, samples=237</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">iops</span><span class="apple-converted-space">
</span><span class="s1">: min=</span><span class="apple-converted-space"> </span>
<span class="s1">228, max= 7970, avg=7029.75, stdev=1044.07, samples=237</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">lat (usec)
</span><span class="apple-converted-space"> </span><span class="s1">: 2=51.56%, 4=47.17%, 10=1.03%, 20=0.22%, 50=0.01%</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">lat (usec)
</span><span class="apple-converted-space"> </span><span class="s1">: 250=0.01%</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">cpu</span><span class="apple-converted-space">
</span><span class="s1">: usr=4.96%, sys=28.37%, ctx=839307, majf=0, minf=26</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">IO depths</span><span class="apple-converted-space">
</span><span class="s1">: 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">submit</span><span class="apple-converted-space">
</span><span class="s1">: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">complete</span><span class="apple-converted-space">
</span><span class="s1">: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">issued rwt: total=839283,838268,0, short=0,0,0, dropped=0,0,0</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">latency
</span><span class="apple-converted-space"> </span><span class="s1">: target=0, window=0, percentile=100.00%, depth=1</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">Run status group 0 (all jobs):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">READ: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3278MiB (3438MB), run=120000-120000msec</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">WRITE: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3274MiB (3434MB), run=120000-120000msec</span><o:p></o:p></p>
<p class="p2"><o:p> </o:p></p>
<p class="p1"><span class="s1">Disk stats (read/write):</span><o:p></o:p></p>
<p class="p1"><span class="apple-converted-space"> </span><span class="s1">nvme0n1: ios=838322/651596, merge=0/0, ticks=83804/22119, in_queue=104773, util=70.18%</span><o:p></o:p></p>
<p class="p1"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Is there a way I can get near bare metal performance from my nvme block devices?<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Arial","sans-serif"">NOTICE<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Arial","sans-serif"">Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright
information. If you are not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability
for the distribution of viruses or similar in electronic communications. This notice should not be removed.
<o:p></o:p></span></p>
</div>
</div>
<div style="font-size:8pt; font-family: Arial">NOTICE</div>
<div style="font-size:8pt; font-family: Arial">Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are
not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution
of viruses or similar in electronic communications. This notice should not be removed.
</div>
</body>
</html>