[openstack-dev] [Neutron][qa] Parallel testing update
enikanorov at mirantis.com
Mon Dec 2 20:24:16 UTC 2013
Salvatore and Armando, thanks for your great work and detailed explanation!
On Mon, Dec 2, 2013 at 11:48 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:
> On Dec 2, 2013 9:04 PM, "Salvatore Orlando" <sorlando at nicira.com> wrote:
> > Hi,
> > As you might have noticed, there has been some progress on parallel
> tests for neutron.
> > In a nutshell:
> > * Armando fixed the issue with IP address exhaustion on the public
> network 
> > * Salvatore has now a patch which has a 50% success rate (the last
> failures are because of me playing with it) 
> > * Salvatore is looking at putting back on track full isolation 
> > * All the bugs affecting parallel tests can be queried here 
> > * This blueprint tracks progress made towards enabling parallel testing
> > ---------
> > The long story is as follows:
> > Parallel testing basically is not working because parallelism means
> higher contention for public IP addresses. This was made worse by the fact
> that some tests created a router with a gateway set but never deleted it.
> As a result, there were even less addresses in the public range.
> >  was already merged and with  we shall make the public network for
> neutron a /24 (the full tempest suite is still showing a lot of IP
> exhaustion errors).
> > However, this was just one part of the issue. The biggest part actually
> lied with the OVS agent and its interactions with the ML2 plugin. A few
> patches (, , ) were already pushed to reduce the number of
> notifications sent from the plugin to the agent. However, the agent is
> organised in a way such that a notification is immediately acted upon thus
> preempting the main agent loop, which is the one responsible for wiring
> ports into networks. Considering the high level of notifications currently
> sent from the server, this becomes particularly wasteful if one consider
> that security membership updates for ports trigger global
> iptables-save/restore commands which are often executed in rapid
> succession, thus resulting in long delays for wiring VIFs to the
> appropriate network.
> > With the patch  we are refactoring the agent to make it more
> efficient. This is not production code, but once we'll get close to 100%
> pass for parallel testing this patch will be split in several patches,
> properly structured, and hopefully easy to review.
> > It is worth noting there is still work to do: in some cases the loop
> still takes too long, and it has been observed ovs commands taking even 10
> seconds to complete. To this aim, it is worth considering use of async
> processes introduced in  as well as leveraging ovsdb monitoring  for
> limiting queries to ovs database.
> > We're still unable to explain some failures where the network appears to
> be correctly wired (floating IP, router port, dhcp port, and VIF port), but
> the SSH connection fails. We're hoping to reproduce this failure patter
> > Finally, the tempest patch for full tempest isolation should be made
> usable soon. Having another experimental job for it is something worth
> considering as for some reason it is not always easy reproducing the same
> failure modes exhibited on the gate.
> > Regards,
> > Salvatore
> Awesome work, thanks for the update.
> >  https://review.openstack.org/#/c/58054/
> >  https://review.openstack.org/#/c/57420/
> >  https://review.openstack.org/#/c/53459/
> >  https://review.openstack.org/#/c/58284/
> >  https://review.openstack.org/#/c/58860/
> >  https://review.openstack.org/#/c/58597/
> >  https://review.openstack.org/#/c/58415/
> >  https://review.openstack.org/#/c/45676/
> >  https://bugs.launchpad.net/neutron/+bug/1177973
> > 
> > 
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the OpenStack-dev