<div dir="ltr">Note that my end goal is to benchmark an application that runs in an instance that does primarily large sequential full-volume-reads.<div><br></div><div>On this path I ran into unexpectedly poor performance within the instance. If this is a common characteristic of OpenStack, then this becomes a question of concern to OpenStack developers.</div><div><br></div><div>Until recently, ~450MB/s would be (and still is for many cases) <i>outstanding</i> throughput. Most similar questions on the web are happy with saturating a couple of gigabit links, or a few spinning disks. So that few(?) folk (to now) have asked questions at this level of performance ... is not a surprise.</div><div><br></div><div>But with flash displacing spinning disks, much higher throughput is possible. If there is an unnecessary bottleneck, this might be a good time to call attention. </div><div><br></div><div><br></div><div>From general notions to current specifics... :)</div><div><br></div><div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 2, 2016 at 10:10 PM, Philipp Marek <span dir="ltr"><<a href="mailto:philipp.marek@linbit.com" target="_blank">philipp.marek@linbit.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">> The benchmark scripts are in:<br>> <a href="https://github.com/pbannister/openstack-bootstrap" rel="noreferrer" target="_blank">https://github.com/pbannister/openstack-bootstrap</a></span></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">in case that might help, here are a few notes and hints about doing<br>
benchmarks for the DRDB block device driver:<br>
<br>
<a href="http://blogs.linbit.com/p/897/benchmarking-drbd/" rel="noreferrer" target="_blank">http://blogs.linbit.com/p/897/benchmarking-drbd/</a><br>
<br>
Perhaps there's something interesting for you.<br></blockquote><div><br></div><div>Found this earlier. :)</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">> Found that if I repeatedly scanned the same 8GB volume from the physical<br>
> host (with 1/4TB of memory), the entire volume was cached in (host) memory<br>
> (very fast scan times).<br></span></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
</span>If the iSCSI target (or QEMU, for direct access) is set up to use buffer<br>
cache, yes.<br>
Whether you really want that is up to discussion - it might be much more<br>
beneficial to move that RAM from the Hypervisor to the VM, which should<br>
then be able to do more efficient caching of the filesystem contents that<br>
it should operate on.<span class=""><br></span></blockquote><div><br></div><div>You are right, but my aim was a bit different. Doing a bit of divide-and-conquer.<br><br>In essence, this test was to see if reducing the host-side volume-read time to (practically) zero would have <i>any</i> impact on performance. Given the <i>huge</i> introduced latency (somewhere), I did not expect a notable difference - and that it what the measure shows. This further supports the theory that host-side Linux is <i>not</i> the issue.</div><div><br></div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
> Scanning the same volume from within the instance still gets the same<br>
> ~450MB/s that I saw before.<br></span></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
</span>Hmmm, with iSCSI inbetween that could be the TCP memcpy limitation.</blockquote><div><br></div><div>Measuring iSCSI in isolation is next on my list. Both on the physical host, and in the instance. (Now to find that link to the iSCSI test, again...)</div><div><br></div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
> The "iostat" numbers from the instance show ~44 %iowait, and ~50 %idle.<br>
> (Which to my reading might explain the ~50% loss of performance.) Why so<br>
> much idle/latency?<br>
><br>
> The in-instance "dd" CPU use is ~12%. (Not very interesting.)<br></span></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
</span>Because your "dd" testcase will be single-threaded, io-depth 1.<br>
And that means synchronous access, each IO has to wait for the preceeding<br>
one to finish...<br></blockquote><div><br></div><div>Given the Linux kernel read-ahead parameter has a noticeable impact on performance, I believe that "dd" does not need wait (much?) for I/O. Note also the large difference between host and instance with "dd".</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">> Not sure from where the (apparent) latency comes. The host iSCSI target?<br>
> The QEMU iSCSI initiator? Onwards...<br></span></blockquote><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
</span>Thread scheduling, inter-CPU cache trashing (if the iSCSI target is on<br>
a different physical CPU package/socket than the VM), ...<br><br>
Benchmarking is a dark art.<br></blockquote><div><br></div><div>This physical host has an absurd number of CPUs (at 40), so what you mention is possible. At these high rates, if only losing 10-20% of the throughput, I might consider such causes. But losing 60% ... my guess ... the cause is much less esoteric.</div><div><br></div><div><br></div><div> </div></div><br></div></div></div>