Open Stack

Salvatore and Armando, thanks for your great work and detailed explanation!

Eugene.


On Mon, Dec 2, 2013 at 11:48 PM, Joe Gordon <joe.gordon0 at gmail.com> wrote:

>
> On Dec 2, 2013 9:04 PM, "Salvatore Orlando" <sorlando at nicira.com> wrote:
> >
> > Hi,
> >
> > As you might have noticed, there has been some progress on parallel
> tests for neutron.
> > In a nutshell:
> > * Armando fixed the issue with IP address exhaustion on the public
> network [1]
> > * Salvatore has now a patch which has a 50% success rate (the last
> failures are because of me playing with it) [2]
> > * Salvatore is looking at putting back on track full isolation [3]
> > * All the bugs affecting parallel tests can be queried here [10]
> > * This blueprint tracks progress made towards enabling parallel testing
> [11]
> >
> > ---------
> > The long story is as follows:
> > Parallel testing basically is not working because parallelism means
> higher contention for public IP addresses. This was made worse by the fact
> that some tests created a router with a gateway set but never deleted it.
> As a result, there were even less addresses in the public range.
> > [1] was already merged and with [4] we shall make the public network for
> neutron a /24 (the full tempest suite is still showing a lot of IP
> exhaustion errors).
> >
> > However, this was just one part of the issue. The biggest part actually
> lied with the OVS agent and its interactions with the ML2 plugin. A few
> patches ([5], [6], [7]) were already pushed to reduce the number of
> notifications sent from the plugin to the agent. However, the agent is
> organised in a way such that a notification is immediately acted upon thus
> preempting the main agent loop, which is the one responsible for wiring
> ports into networks. Considering the high level of notifications currently
> sent from the server, this becomes particularly wasteful if one consider
> that security membership updates for ports trigger global
> iptables-save/restore commands which are often executed in rapid
> succession, thus resulting in long delays for wiring VIFs to the
> appropriate network.
> > With the patch [2] we are refactoring the agent to make it more
> efficient. This is not production code, but once we'll get close to 100%
> pass for parallel testing this patch will be split in several patches,
> properly structured, and hopefully easy to review.
> > It is worth noting there is still work to do: in some cases the loop
> still takes too long, and it has been observed ovs commands taking even 10
> seconds to complete. To this aim, it is worth considering use of async
> processes introduced in [8] as well as leveraging ovsdb monitoring [9] for
> limiting queries to ovs database.
> > We're still unable to explain some failures where the network appears to
> be correctly wired (floating IP, router port, dhcp port, and VIF port), but
> the SSH connection fails. We're hoping to reproduce this failure patter
> locally.
> >
> > Finally, the tempest patch for full tempest isolation should be made
> usable soon. Having another experimental job for it is something worth
> considering as for some reason it is not always easy reproducing the same
> failure modes exhibited on the gate.
> >
> > Regards,
> > Salvatore
> >
>
> Awesome work, thanks for the update.
>
> > [1] https://review.openstack.org/#/c/58054/
> > [2] https://review.openstack.org/#/c/57420/
> > [3] https://review.openstack.org/#/c/53459/
> > [4] https://review.openstack.org/#/c/58284/
> > [5] https://review.openstack.org/#/c/58860/
> > [6] https://review.openstack.org/#/c/58597/
> > [7] https://review.openstack.org/#/c/58415/
> > [8] https://review.openstack.org/#/c/45676/
> > [9] https://bugs.launchpad.net/neutron/+bug/1177973
> > [10]
> https://bugs.launchpad.net/neutron/+bugs?field.tag=neutron-parallel&field.tags_combinator=ANY
> > [11]
> https://blueprints.launchpad.net/neutron/+spec/neutron-tempest-parallel
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131203/a261e53d/attachment.html>