<div dir="ltr">And you can use the nova option "<span style="color:rgb(24,54,145);font-family:consolas,"liberation mono",menlo,courier,monospace;font-size:12px;line-height:20px;white-space:pre;background-color:rgb(248,238,199)">live_migration_inbound_addr</span>" [1] to avoid the impact of live migration at all, but that require additional network plane.<div><br></div><div>[1]: <a href="https://github.com/openstack/nova/blob/master/nova/conf/libvirt.py#L243-L258">https://github.com/openstack/nova/blob/master/nova/conf/libvirt.py#L243-L258</a><br></div><div>[2]: <a href="https://blueprints.launchpad.net/nova/+spec/split-network-plane-for-live-migration">https://blueprints.launchpad.net/nova/+spec/split-network-plane-for-live-migration</a></div></div><div class="gmail_extra"><br><div class="gmail_quote">2016-12-30 20:56 GMT+08:00 Koniszewski, Pawel <span dir="ltr"><<a href="mailto:pawel.koniszewski@intel.com" target="_blank">pawel.koniszewski@intel.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div class="m_-7687117907456739778WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Matt,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">This was a bandwidth issue. Nova kept being connected to the broker, but it started to timeout on RPC messages, e.g., we lost some RPC messages triggering post
live migration steps which are there to update nova DB to reflect new host of an instance.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">The good workaround for this issue is to slightly limit bandwidth used for live migrations through nova.conf [1], live_migration_bandwidth config option in [libvirt]
section. By default it is set to 0, so, basically, it is unlimited. Also please be aware that we changed default live migration configuration in OpenStack Newton and tunneling is now off by default (live_migration_tunnelled in [libvirt] section is set to False)
due to huge performance impact.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">[1]
<a href="http://docs.openstack.org/newton/config-reference/compute/config-options.html" target="_blank">
http://docs.openstack.org/<wbr>newton/config-reference/<wbr>compute/config-options.html</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Kind Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">Pawel Koniszewski<u></u><u></u></span></p>
<p class="MsoNormal"><a name="m_-7687117907456739778__MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></a></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a name="m_-7687117907456739778______replyseparator"></a><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <a href="mailto:tadowguy@gmail.com" target="_blank">tadowguy@gmail.com</a> [mailto:<a href="mailto:tadowguy@gmail.com" target="_blank">tadowguy@gmail.com</a>]
<b>On Behalf Of </b>Matt Fischer<br>
<b>Sent:</b> Friday, December 30, 2016 5:31 AM<br>
<b>To:</b> Koniszewski, Pawel <<a href="mailto:pawel.koniszewski@intel.com" target="_blank">pawel.koniszewski@intel.com</a>><br>
<b>Cc:</b> <a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.<wbr>openstack.org</a><br>
<b>Subject:</b> Re: [Openstack-operators] [nova] Live migration performance tests on 100 compute nodes<u></u><u></u></span></p>
</div>
</div><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">On Wed, Dec 28, 2016 at 6:11 AM, Koniszewski, Pawel <<a href="mailto:pawel.koniszewski@intel.com" target="_blank">pawel.koniszewski@intel.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Hello everyone,<br>
<br>
We made a research to see how live migration performance varies between different configurations, especially we aimed to test tunneled vs non-tunneled live migrations. To test live migration we simulated a case of 0-day patching of 100 compute nodes (including
reboot) with workloads that are close to the real world workloads. All the results were published [1] along with environment configuration and how we built test framework. Hope you find this useful.<br>
<br>
[1] <a href="https://01.org/openstack/blogs/pkoniszewski/2016/ossc-zero-day-patching" target="_blank">
https://01.org/openstack/<wbr>blogs/pkoniszewski/2016/ossc-<wbr>zero-day-patching</a><br>
<br>
Kind Regards,<br>
Pawel Koniszewski<u></u><u></u></p>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks for the write-up. I'm curious about your RabbitMQ connection failures. Was it nova-compute failing to connect? Was it a bandwidth or heartbeat issue?<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div></div></div>
</div>
</div>
<br>______________________________<wbr>_________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.<wbr>openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-operators</a><br>
<br></blockquote></div><br></div>