<div dir="ltr">I have already a patch under review for the quota test, for which I adopted the shortest-diff approach.<div>As regards Robert's suggestion, the problem we have there is that the test uses a dedicated tenant, but it does not take into account the possibility that at some point the dhcp agent will create a port too for that tenant.</div>
<div><br></div><div>In theory I tend to agree with Miguel; but I'm not sure what would be the consensus on removing a scenario test. I think we either decide to merge this shortest-diff patch [1] once the comments are addressed, or re-design the tests, which might take some more time.</div>
<div><br></div><div>Salvatore</div><div><br></div><div>PS: shortest-diff is, as you might have already understood, an euphemism for 'hack'</div><div><br></div><div><br></div><div>[1] <a href="https://review.openstack.org/#/c/64217/">https://review.openstack.org/#/c/64217/</a></div>
<div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 2 January 2014 22:39, Robert Collins <span dir="ltr"><<a href="mailto:robertc@robertcollins.net" target="_blank">robertc@robertcollins.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Another way to tackle it would be to create a dedicated tenant for<br>
those tests, then the quota won't interact with anything else.<br>
<div class="HOEnZb"><div class="h5"><br>
On 3 January 2014 10:35, Miguel Angel Ajo Pelayo <<a href="mailto:mangelajo@redhat.com">mangelajo@redhat.com</a>> wrote:<br>
> Hi Salvatore!,<br>
><br>
> Good work on this.<br>
><br>
> About the quota limit tests, I believe they may be unit-tested,<br>
> instead of functionally tested.<br>
><br>
> When running those tests in parallel with any other tests that rely<br>
> on having ports, networks or subnets available into quota, they have<br>
> high chances of making those other tests fail.<br>
><br>
> Cheers,<br>
> Miguel Ángel Ajo<br>
><br>
><br>
><br>
> ----- Original Message -----<br>
>> From: "Kyle Mestery" <<a href="mailto:mestery@siliconloons.com">mestery@siliconloons.com</a>><br>
>> To: "OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
>> Sent: Thursday, January 2, 2014 7:53:05 PM<br>
>> Subject: Re: [openstack-dev] [Neutron][qa] Parallel testing update<br>
>><br>
>> Thanks for the updates here Salvatore, and for continuing to push on<br>
>> this! This is all great work!<br>
>><br>
>> On Jan 2, 2014, at 6:57 AM, Salvatore Orlando <<a href="mailto:sorlando@nicira.com">sorlando@nicira.com</a>> wrote:<br>
>> ><br>
>> > Hi again,<br>
>> ><br>
>> > I've now run the experimental job a good deal of times, and I've filed bugs<br>
>> > for all the issues which came out.<br>
>> > Most of them occurred no more than once among all test execution (I think<br>
>> > about 30).<br>
>> ><br>
>> > They're all tagged with neutron-parallel [1]. for ease of tracking, I've<br>
>> > associated all the bug reports with neutron, but some are probably more<br>
>> > tempest or nova issues.<br>
>> ><br>
>> > Salvatore<br>
>> ><br>
>> > [1] <a href="https://bugs.launchpad.net/neutron/+bugs?field.tag=neutron-parallel" target="_blank">https://bugs.launchpad.net/neutron/+bugs?field.tag=neutron-parallel</a><br>
>> ><br>
>> ><br>
>> > On 27 December 2013 11:09, Salvatore Orlando <<a href="mailto:sorlando@nicira.com">sorlando@nicira.com</a>> wrote:<br>
>> > Hi,<br>
>> ><br>
>> > We now have several patches under review which improve a lot how neutron<br>
>> > handles parallel testing.<br>
>> > In a nutshell, these patches try to ensure the ovs agent processes new,<br>
>> > removed, and updated interfaces as soon as possible,<br>
>> ><br>
>> > These patches are:<br>
>> > <a href="https://review.openstack.org/#/c/61105/" target="_blank">https://review.openstack.org/#/c/61105/</a><br>
>> > <a href="https://review.openstack.org/#/c/61964/" target="_blank">https://review.openstack.org/#/c/61964/</a><br>
>> > <a href="https://review.openstack.org/#/c/63100/" target="_blank">https://review.openstack.org/#/c/63100/</a><br>
>> > <a href="https://review.openstack.org/#/c/63558/" target="_blank">https://review.openstack.org/#/c/63558/</a><br>
>> ><br>
>> > There is still room for improvement. For instance the calls from the agent<br>
>> > into the plugins might be consistently reduced.<br>
>> > However, even if the above patches shrink a lot the time required for<br>
>> > processing a device, we are still hitting a hard limit with the execution<br>
>> > ovs commands for setting local vlan tags and clearing flows (or adding the<br>
>> > flow rule for dropping all the traffic).<br>
>> > In some instances this commands slow down a lot, requiring almost 10<br>
>> > seconds to complete. This adds a delay in interface processing which in<br>
>> > some cases leads to the hideous SSH timeout error (the same we see with<br>
>> > bug 1253896 in normal testing).<br>
>> > It is also worth noting that when this happens sysstat reveal CPU usage is<br>
>> > very close to 100%<br>
>> ><br>
>> > From the neutron side there is little we can do. Introducing parallel<br>
>> > processing for interface, as we do for the l3 agent, is not actually a<br>
>> > solution, since ovs-vswitchd v1.4.x, the one executed on gate tests, is<br>
>> > not multithreaded. If you think the situation might be improved by<br>
>> > changing the logic for handling local vlan tags and putting ports on the<br>
>> > dead vlan, I would be happy to talk about that.<br>
>> > On my local machines I've seen a dramatic improvement in processing times<br>
>> > by installing ovs 2.0.0, which has a multi-threaded vswitchd. Is this<br>
>> > something we might consider for gate tests? Also, in order to reduce CPU<br>
>> > usage on the gate (and making tests a bit faster), there is a tempest<br>
>> > patch which stops creating and wiring neutron routers when they're not<br>
>> > needed: <a href="https://review.openstack.org/#/c/62962/" target="_blank">https://review.openstack.org/#/c/62962/</a><br>
>> ><br>
>> > Even in my local setup which succeeds about 85% of times, I'm still seeing<br>
>> > some occurrences of the issue described in [1], which at the end of the<br>
>> > day seems a dnsmasq issue.<br>
>> ><br>
>> > Beyond the 'big' structural problem discussed above, there are some minor<br>
>> > problems with a few tests:<br>
>> ><br>
>> > 1) test_network_quotas.test_create_ports_until_quota_hit fails about 90%<br>
>> > of times. I think this is because the test itself should be made aware of<br>
>> > parallel execution and asynchronous events, and there is a patch for this<br>
>> > already: <a href="https://review.openstack.org/#/c/64217" target="_blank">https://review.openstack.org/#/c/64217</a><br>
>> ><br>
>> > 2) test_attach_interfaces.test_create_list_show_delete_interfaces fails<br>
>> > about 66% of times. The failure is always on an assertion made after<br>
>> > deletion of interfaces, which probably means the interface is not deleted<br>
>> > within 5 seconds. I think this might be a consequence of the higher load<br>
>> > on the neutron service and we might try to enable multiple workers on the<br>
>> > gate to this aim, or just increase the tempest timeout. On a slightly<br>
>> > different note, allow me to say that the way assertion are made on this<br>
>> > test might be improved a bit. So far one has to go through the code to see<br>
>> > why the test failed.<br>
>> ><br>
>> > Thanks for reading this rather long message.<br>
>> > Regards,<br>
>> > Salvatore<br>
>> ><br>
>> > [1] <a href="https://lists.launchpad.net/openstack/msg23817.html" target="_blank">https://lists.launchpad.net/openstack/msg23817.html</a><br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > On 2 December 2013 22:01, Kyle Mestery (kmestery) <<a href="mailto:kmestery@cisco.com">kmestery@cisco.com</a>><br>
>> > wrote:<br>
>> > Yes, this is all great Salvatore and Armando! Thank you for all of this<br>
>> > work<br>
>> > and the explanation behind it all.<br>
>> ><br>
>> > Kyle<br>
>> ><br>
>> > On Dec 2, 2013, at 2:24 PM, Eugene Nikanorov <<a href="mailto:enikanorov@mirantis.com">enikanorov@mirantis.com</a>><br>
>> > wrote:<br>
>> ><br>
>> > > Salvatore and Armando, thanks for your great work and detailed<br>
>> > > explanation!<br>
>> > ><br>
>> > > Eugene.<br>
>> > ><br>
>> > ><br>
>> > > On Mon, Dec 2, 2013 at 11:48 PM, Joe Gordon <<a href="mailto:joe.gordon0@gmail.com">joe.gordon0@gmail.com</a>><br>
>> > > wrote:<br>
>> > ><br>
>> > > On Dec 2, 2013 9:04 PM, "Salvatore Orlando" <<a href="mailto:sorlando@nicira.com">sorlando@nicira.com</a>> wrote:<br>
>> > > ><br>
>> > > > Hi,<br>
>> > > ><br>
>> > > > As you might have noticed, there has been some progress on parallel<br>
>> > > > tests for neutron.<br>
>> > > > In a nutshell:<br>
>> > > > * Armando fixed the issue with IP address exhaustion on the public<br>
>> > > > network [1]<br>
>> > > > * Salvatore has now a patch which has a 50% success rate (the last<br>
>> > > > failures are because of me playing with it) [2]<br>
>> > > > * Salvatore is looking at putting back on track full isolation [3]<br>
>> > > > * All the bugs affecting parallel tests can be queried here [10]<br>
>> > > > * This blueprint tracks progress made towards enabling parallel testing<br>
>> > > > [11]<br>
>> > > ><br>
>><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
<br>
</div></div><span class="HOEnZb"><font color="#888888">--<br>
Robert Collins <<a href="mailto:rbtcollins@hp.com">rbtcollins@hp.com</a>><br>
Distinguished Technologist<br>
HP Converged Cloud<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>