<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#232729" text="#eeeeec" link="#4a90d9" vlink="#eeeeec">
<div>Hi,</div>
<div><br>
</div>
<div>just to let you know. Problem is now gone. Instances boot up with working network interface.</div>
<div><br>
</div>
<div>Thanks a lot,</div>
<div>Radu</div>
<div><br>
</div>
<div>On Tue, 2018-05-29 at 21:23 -0400, Chris Apsey wrote:</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div dir="auto">I want to echo the effectiveness of this change - we had vif failures when launching more than 50 or so cirros instances simultaneously, but moving to daemon mode made this issue disappear and we've tested 5x that amount. This has been the
single biggest scalability improvement to date. This option should be the default in the official docs.</div>
<div dir="auto"><br>
</div>
<div id="aqm-original" style="font-family: sans-serif; font-size: 12pt; color: black;">
<!-- body start -->
<div class="aqm-original-body">
<div style="color: black;">
<p style="color: black; font-size: 10pt; font-family: Arial, sans-serif; margin: 8pt 0;">
On May 24, 2018 05:55:49 Saverio Proto <zioproto@gmail.com> wrote:</p>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div dir="auto">Glad to hear it!
<div dir="auto">Always monitor rabbitmq queues to identify bottlenecks !! :)</div>
<div dir="auto"><br>
</div>
<div dir="auto">Cheers</div>
<div dir="auto"><br>
</div>
<div dir="auto">Saverio</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr">Il gio 24 mag 2018, 11:07 Radu Popescu | eMAG, Technology <<a href="mailto:radu.popescu@emag.ro">radu.popescu@emag.ro</a>> ha scritto:<br>
</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div bgcolor="#232729" text="#eeeeec" link="#4a90d9" vlink="#eeeeec">
<div>Hi,</div>
<div><br>
</div>
<div>did the change yesterday. Had no issue this morning with neutron not being able to move fast enough. Still, we had some storage issues, but that's another thing.</div>
<div>Anyway, I'll leave it like this for the next few days and report back in case I get the same slow neutron errors.</div>
<div><br>
</div>
<div>Thanks a lot!</div>
<div>Radu</div>
<div><br>
</div>
<div>On Wed, 2018-05-23 at 10:08 +0000, Radu Popescu | eMAG, Technology wrote:</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<div>Hi,</div>
<div><br>
</div>
<div>actually, I didn't know about that option. I'll enable it right now.</div>
<div>Testing is done every morning at about 4:00AM ..so I'll know tomorrow morning if it changed anything.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Radu</div>
<div><br>
</div>
<div>On Tue, 2018-05-22 at 15:30 +0200, Saverio Proto wrote:</div>
<blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex">
<pre>Sorry email went out incomplete.</pre>
<pre>Read this:</pre>
<pre><a href="https://cloudblog.switch.ch/2017/08/28/starting-1000-instances-on-switchengines/" target="_blank" rel="noreferrer">https://cloudblog.switch.ch/2017/08/28/starting-1000-instances-on-switchengines/</a></pre>
<pre><br></pre>
<pre>make sure that Openstack rootwrap configured to work in daemon mode</pre>
<pre><br></pre>
<pre>Thank you</pre>
<pre><br></pre>
<pre>Saverio</pre>
<pre><br></pre>
<pre><br></pre>
<pre>2018-05-22 15:29 GMT+02:00 Saverio Proto <<a href="mailto:zioproto@gmail.com" target="_blank" rel="noreferrer">zioproto@gmail.com</a>>:</pre>
<pre><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"></blockquote></pre>
<pre>Hello Radu,</pre>
<pre><br></pre>
<pre>do you have the Openstack rootwrap configured to work in daemon mode ?</pre>
<pre><br></pre>
<pre>please read this article:</pre>
<pre><br></pre>
<pre>2018-05-18 10:21 GMT+02:00 Radu Popescu | eMAG, Technology</pre>
<pre><<a href="mailto:radu.popescu@emag.ro" target="_blank" rel="noreferrer">radu.popescu@emag.ro</a>>:</pre>
<pre><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"></blockquote></pre>
<pre>Hi,</pre>
<pre><br></pre>
<pre>so, nova says the VM is ACTIVE and actually boots with no network. We are</pre>
<pre>setting some metadata that we use later on and have cloud-init for different</pre>
<pre>tasks.</pre>
<pre>So, VM is up, OS is running, but network is working after a random amount of</pre>
<pre>time, that can get to around 45 minutes. Thing is, is not happening to all</pre>
<pre>VMs in that test (around 300), but it's happening to a fair amount - around</pre>
<pre>25%.</pre>
<pre><br></pre>
<pre>I can see the callback coming few seconds after neutron openvswitch agent</pre>
<pre>says it's completed the setup. My question is, why is it taking so long for</pre>
<pre>nova openvswitch agent to configure the port? I can see the port up in both</pre>
<pre>host OS and openvswitch. I would assume it's doing the whole namespace and</pre>
<pre>iptables setup. But still, 30 minutes? Seems a lot!</pre>
<pre><br></pre>
<pre>Thanks,</pre>
<pre>Radu</pre>
<pre><br></pre>
<pre>On Thu, 2018-05-17 at 11:50 -0400, George Mihaiescu wrote:</pre>
<pre><br></pre>
<pre>We have other scheduled tests that perform end-to-end (assign floating IP,</pre>
<pre>ssh, ping outside) and never had an issue.</pre>
<pre>I think we turned it off because the callback code was initially buggy and</pre>
<pre>nova would wait forever while things were in fact ok, but I'll change</pre>
<pre>"vif_plugging_is_fatal = True" and "vif_plugging_timeout = 300" and run</pre>
<pre>another large test, just to confirm.</pre>
<pre><br></pre>
<pre>We usually run these large tests after a version upgrade to test the APIs</pre>
<pre>under load.</pre>
<pre><br></pre>
<pre><br></pre>
<pre><br></pre>
<pre>On Thu, May 17, 2018 at 11:42 AM, Matt Riedemann <<a href="mailto:mriedemos@gmail.com" target="_blank" rel="noreferrer">mriedemos@gmail.com</a>></pre>
<pre>wrote:</pre>
<pre><br></pre>
<pre>On 5/17/2018 9:46 AM, George Mihaiescu wrote:</pre>
<pre><br></pre>
<pre>and large rally tests of 500 instances complete with no issues.</pre>
<pre><br></pre>
<pre><br></pre>
<pre>Sure, except you can't ssh into the guests.</pre>
<pre><br></pre>
<pre>The whole reason the vif plugging is fatal and timeout and callback code was</pre>
<pre>because the upstream CI was unstable without it. The server would report as</pre>
<pre>ACTIVE but the ports weren't wired up so ssh would fail. Having an ACTIVE</pre>
<pre>guest that you can't actually do anything with is kind of pointless.</pre>
<pre><br></pre>
<pre>_______________________________________________</pre>
<pre><br></pre>
<pre>OpenStack-operators mailing list</pre>
<pre><br></pre>
<pre><a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank" rel="noreferrer">OpenStack-operators@lists.openstack.org</a></pre>
<pre><br></pre>
<pre><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank" rel="noreferrer">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a></pre>
<pre><br></pre>
<pre><br></pre>
<pre><br></pre>
<pre>_______________________________________________</pre>
<pre>OpenStack-operators mailing list</pre>
<pre><a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank" rel="noreferrer">OpenStack-operators@lists.openstack.org</a></pre>
<pre><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank" rel="noreferrer">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a></pre>
<pre><br></pre>
<pre></pre>
</blockquote>
<pre>_______________________________________________</pre>
<pre>OpenStack-operators mailing list</pre>
<pre><a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank" rel="noreferrer">OpenStack-operators@lists.openstack.org</a></pre>
<pre><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank" rel="noreferrer">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a></pre>
<pre><br></pre>
</blockquote>
</div>
</blockquote>
</div>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a class="aqm-autolink aqm-autowrap" href="mailto:OpenStack-operators%40lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a class="aqm-autolink aqm-autowrap" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
<br>
</blockquote>
</div>
</div>
<!-- body end --></div>
<div dir="auto"><br>
</div>
</blockquote>
</body>
</html>