<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" id="owaParaStyle"></style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Dear Openstack community,
<div><br>
</div>
<div>I would like to have a high performance distributed database running in Openstack vms. I tried attaching dedicated nvme pci devices to the vm but the performance is not as good as I can get from bare metal.</div>
<div><br>
</div>
<div>Bare metal:</div>
<div><br>
</div>
<div><style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; min-height: 13.0px}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style>
<p class="p1"><span class="s1">[root@zeus-54 data]# fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based</span></p>
<p class="p1"><span class="s1">test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1</span></p>
<p class="p1"><span class="s1">fio-3.1</span></p>
<p class="p1"><span class="s1">Starting 1 process</span></p>
<p class="p1"><span class="s1">Jobs: 1 (f=1): [f(1)][100.0%][r=39.5MiB/s,w=39.6MiB/s][r=10.1k,w=10.1k IOPS][eta 00m:00s]</span></p>
<p class="p1"><span class="s1">test: (groupid=0, jobs=1): err= 0: pid=50892: Wed May<span class="Apple-converted-space">
</span>1 22:22:45 2019</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>read: IOPS=9805, BW=38.3MiB/s (40.2MB/s)(4596MiB/120001msec)</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>slat (usec): min=39, max=6678, avg=94.72, stdev=55.78</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>clat (nsec): min=450, max=18224, avg=525.83, stdev=120.10</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>lat (usec): min=39, max=6679, avg=95.36, stdev=55.79</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>clat percentiles (nsec):</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>|<span class="Apple-converted-space">
</span>1.00th=[<span class="Apple-converted-space"> </span>462],<span class="Apple-converted-space">
</span>5.00th=[<span class="Apple-converted-space"> </span>478], 10.00th=[<span class="Apple-converted-space">
</span>482], 20.00th=[<span class="Apple-converted-space"> </span>486],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 30.00th=[<span class="Apple-converted-space">
</span>490], 40.00th=[<span class="Apple-converted-space"> </span>494], 50.00th=[<span class="Apple-converted-space">
</span>502], 60.00th=[<span class="Apple-converted-space"> </span>510],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 70.00th=[<span class="Apple-converted-space">
</span>516], 80.00th=[<span class="Apple-converted-space"> </span>532], 90.00th=[<span class="Apple-converted-space">
</span>596], 95.00th=[<span class="Apple-converted-space"> </span>676],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 99.00th=[<span class="Apple-converted-space">
</span>860], 99.50th=[ 1048], 99.90th=[ 1384], 99.95th=[ 2480],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 99.99th=[ 3728]</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>bw (<span class="Apple-converted-space">
</span>KiB/s): min=<span class="Apple-converted-space"> </span>720, max=40736, per=100.00%, avg=39389.00, stdev=5317.58, samples=239</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>iops<span class="Apple-converted-space">
</span>: min=<span class="Apple-converted-space"> </span>180, max=10184, avg=9847.23, stdev=1329.39, samples=239</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>write: IOPS=9799, BW=38.3MiB/s (40.1MB/s)(4594MiB/120001msec)</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>slat (nsec): min=2982, max=106207, avg=4220.09, stdev=980.04</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>clat (nsec): min=407, max=18130, avg=451.48, stdev=103.71</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>lat (usec): min=3, max=111, avg= 4.74, stdev= 1.03</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>clat percentiles (nsec):</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>|<span class="Apple-converted-space">
</span>1.00th=[<span class="Apple-converted-space"> </span>414],<span class="Apple-converted-space">
</span>5.00th=[<span class="Apple-converted-space"> </span>418], 10.00th=[<span class="Apple-converted-space">
</span>422], 20.00th=[<span class="Apple-converted-space"> </span>430],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 30.00th=[<span class="Apple-converted-space">
</span>434], 40.00th=[<span class="Apple-converted-space"> </span>434], 50.00th=[<span class="Apple-converted-space">
</span>438], 60.00th=[<span class="Apple-converted-space"> </span>438],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 70.00th=[<span class="Apple-converted-space">
</span>442], 80.00th=[<span class="Apple-converted-space"> </span>446], 90.00th=[<span class="Apple-converted-space">
</span>462], 95.00th=[<span class="Apple-converted-space"> </span>588],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 99.00th=[<span class="Apple-converted-space">
</span>700], 99.50th=[<span class="Apple-converted-space"> </span>916], 99.90th=[ 1208], 99.95th=[ 1288],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 99.99th=[ 3536]</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>bw (<span class="Apple-converted-space">
</span>KiB/s): min=<span class="Apple-converted-space"> </span>752, max=42608, per=100.00%, avg=39366.63, stdev=5355.73, samples=239</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>iops<span class="Apple-converted-space">
</span>: min=<span class="Apple-converted-space"> </span>188, max=10652, avg=9841.64, stdev=1338.93, samples=239</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>lat (nsec)
<span class="Apple-converted-space"> </span>: 500=69.98%, 750=28.64%, 1000=0.90%</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>lat (usec)
<span class="Apple-converted-space"> </span>: 2=0.42%, 4=0.04%, 10=0.01%, 20=0.01%</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>cpu<span class="Apple-converted-space">
</span>: usr=2.20%, sys=10.85%, ctx=1176675, majf=0, minf=1372</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>IO depths<span class="Apple-converted-space">
</span>: 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>submit<span class="Apple-converted-space">
</span>: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>complete<span class="Apple-converted-space">
</span>: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>issued rwt: total=1176625,1175958,0, short=0,0,0, dropped=0,0,0</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>latency
<span class="Apple-converted-space"> </span>: target=0, window=0, percentile=100.00%, depth=1</span></p>
<p class="p2"><span class="s1"></span><br>
</p>
<p class="p1"><span class="s1">Run status group 0 (all jobs):</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>READ: bw=38.3MiB/s (40.2MB/s), 38.3MiB/s-38.3MiB/s (40.2MB/s-40.2MB/s), io=4596MiB (4819MB), run=120001-120001msec</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>WRITE: bw=38.3MiB/s (40.1MB/s), 38.3MiB/s-38.3MiB/s (40.1MB/s-40.1MB/s), io=4594MiB (4817MB), run=120001-120001msec</span></p>
<p class="p2"><span class="s1"></span><br>
</p>
<p class="p1"><span class="s1">Disk stats (read/write):</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>nvme9n1: ios=1174695/883620, merge=0/0, ticks=105502/72225, in_queue=192101, util=99.28%</span></p>
<p class="p1"><span class="s1"><br>
</span></p>
<p class="p1"><span class="s1">From vm:</span></p>
<p class="p1"><span class="s1">[centos@kudu-1 nvme0]$ sudo fio --ioengine=libaio --name=test --filename=test --bs=4k --size=40G --readwrite=randrw --runtime=120 --time_based</span></p>
<p class="p1"><span class="s1">test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=1</span></p>
<p class="p1"><span class="s1">fio-3.1</span></p>
<p class="p1"><span class="s1">Starting 1 process</span></p>
<p class="p1"><span class="s1">Jobs: 1 (f=1): [m(1)][100.0%][r=29.2MiB/s,w=29.7MiB/s][r=7487,w=7595 IOPS][eta 00m:00s]</span></p>
<p class="p1"><span class="s1">test: (groupid=0, jobs=1): err= 0: pid=44383: Wed May<span class="Apple-converted-space">
</span>1 12:22:24 2019</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>read: IOPS=6994, BW=27.3MiB/s (28.6MB/s)(3278MiB/120000msec)</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>slat (usec): min=54, max=20476, avg=115.27, stdev=71.45</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>clat (nsec): min=1757, max=31476, avg=2163.02, stdev=688.66</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>lat (usec): min=56, max=20481, avg=118.51, stdev=71.66</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>clat percentiles (nsec):</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>|<span class="Apple-converted-space">
</span>1.00th=[ 1800],<span class="Apple-converted-space"> </span>5.00th=[ 1832], 10.00th=[ 1864], 20.00th=[ 1992],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 30.00th=[ 2040], 40.00th=[ 2064], 50.00th=[ 2064], 60.00th=[ 2096],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 70.00th=[ 2096], 80.00th=[ 2128], 90.00th=[ 2480], 95.00th=[ 2544],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 99.00th=[ 4448], 99.50th=[ 5536], 99.90th=[11072], 99.95th=[12736],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 99.99th=[18560]</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>bw (<span class="Apple-converted-space">
</span>KiB/s): min=<span class="Apple-converted-space"> </span>952, max=31224, per=100.00%, avg=28153.51, stdev=4126.89, samples=237</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>iops<span class="Apple-converted-space">
</span>: min=<span class="Apple-converted-space"> </span>238, max= 7806, avg=7038.23, stdev=1031.70, samples=237</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>write: IOPS=6985, BW=27.3MiB/s (28.6MB/s)(3274MiB/120000msec)</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>slat (usec): min=7, max=963, avg=12.60, stdev= 6.24</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>clat (nsec): min=1662, max=199250, avg=2030.26, stdev=712.33</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>lat (usec): min=10, max=970, avg=15.68, stdev= 6.48</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>clat percentiles (nsec):</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>|<span class="Apple-converted-space">
</span>1.00th=[ 1688],<span class="Apple-converted-space"> </span>5.00th=[ 1720], 10.00th=[ 1736], 20.00th=[ 1864],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 30.00th=[ 1928], 40.00th=[ 1944], 50.00th=[ 1944], 60.00th=[ 1960],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 70.00th=[ 1960], 80.00th=[ 1992], 90.00th=[ 2352], 95.00th=[ 2384],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 99.00th=[ 4048], 99.50th=[ 4768], 99.90th=[11456], 99.95th=[13120],</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>| 99.99th=[19072]</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>bw (<span class="Apple-converted-space">
</span>KiB/s): min=<span class="Apple-converted-space"> </span>912, max=31880, per=100.00%, avg=28119.64, stdev=4176.38, samples=237</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>iops<span class="Apple-converted-space">
</span>: min=<span class="Apple-converted-space"> </span>228, max= 7970, avg=7029.75, stdev=1044.07, samples=237</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>lat (usec)
<span class="Apple-converted-space"> </span>: 2=51.56%, 4=47.17%, 10=1.03%, 20=0.22%, 50=0.01%</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>lat (usec)
<span class="Apple-converted-space"> </span>: 250=0.01%</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>cpu<span class="Apple-converted-space">
</span>: usr=4.96%, sys=28.37%, ctx=839307, majf=0, minf=26</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>IO depths<span class="Apple-converted-space">
</span>: 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>submit<span class="Apple-converted-space">
</span>: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>complete<span class="Apple-converted-space">
</span>: 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>issued rwt: total=839283,838268,0, short=0,0,0, dropped=0,0,0</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>latency
<span class="Apple-converted-space"> </span>: target=0, window=0, percentile=100.00%, depth=1</span></p>
<p class="p2"><span class="s1"></span><br>
</p>
<p class="p1"><span class="s1">Run status group 0 (all jobs):</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>READ: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3278MiB (3438MB), run=120000-120000msec</span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>WRITE: bw=27.3MiB/s (28.6MB/s), 27.3MiB/s-27.3MiB/s (28.6MB/s-28.6MB/s), io=3274MiB (3434MB), run=120000-120000msec</span></p>
<p class="p2"><span class="s1"></span><br>
</p>
<p class="p1"><span class="s1">Disk stats (read/write):</span></p>
<p class="p1"><span class="s1"><style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; min-height: 13.0px}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style></span></p>
<p class="p1"><span class="s1"><span class="Apple-converted-space"> </span>nvme0n1: ios=838322/651596, merge=0/0, ticks=83804/22119, in_queue=104773, util=70.18%</span></p>
<p class="p1"><span class="s1"><br>
</span></p>
</div>
<div>Is there a way I can get near bare metal performance from my nvme block devices?</div>
</div>
<div style="font-size:8pt; font-family: Arial">NOTICE</div>
<div style="font-size:8pt; font-family: Arial">Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are
not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution
of viruses or similar in electronic communications. This notice should not be removed.
</div>
</body>
</html>