[openstack-dev] [neutron][upgrades] Potential issues when performing Neutron upgrades

Ihar Hrachyshka ihrachys at redhat.com
Thu Jul 9 14:11:00 UTC 2015


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On 07/09/2015 09:01 AM, Artur Korzen wrote:
> Hi all,
> 
> I've been researching the Neutron project as a part of work on
> Openstack rolling upgrades, my primary assignments included testing
> if there is no VM access downtime when performing upgrade.
> 
> 
> 
> Are you aware of any issues that are present in Liberty release of 
> Neutron, that may cause the VM access downtime and may stop ops to
> pick up newest versions of code?
> 
> 
> 
> When talking about no VM access downtime during upgrade, the
> sensitive operations are:
> 
> 1. ensure that even after neutron services is shutoff/uninstalled,
> the traffic can go into the VM 2. take care on neutron services
> startup/installation that existing configuration is preserved
> (routing tables, forwarding entries and security rules) 3. the
> underlying network technologies (OVS, linux bridges etc.) is 
> working without distraction when upgrading neutron code: no
> dropping table flows or applying the traffic rules to drop the
> packages
> 
> 
> 
> To be compatible between different versions of service talking to
> each other, the important areas are:
> 
> 1. RPC API versioning 2. Objects exchanged between services via
> RPC 3. Database schema (bp [2]) and data migration
> 
> 
> 
> I have researched the ML2 plugin with ovs agent, including the 
> neutron-ovs-agent, dhcp-server, l3-agent, neutron-server.
> 
> One of the VM access downtime during upgrade is addressed in [1], 
> removing the flows in ovs after neutron agent restarts.
> 
> 
> 
> Is Neutron ready to be upgradable with minimal downtime of services
> and no VM access downtime?

As the ovs bug you refer to above, no, at least not in reference
implementation. That's for data plane.

As for other services, neutron-server online schema migration should
help, and I hope to get it implemented in L (though there are some
obstacles that may block us; we'll see).

As for live data migration, neutron code base is not ready yet to
reasonably require live data migration being implemented in all
patches that need data moves in database. The very first obstacle for
that is that there is no middleware layer between sqlalchemy and the
rest of neutron that would allow us to hide migration details.
Oslo.versionedobjects is such a middleware. See below.

> 
> Are there any guidelines on using the oslo versionobjects and its 
> priority in Liberty cycle?

It's not a common priority for L, but we've started on this road
inside feature/qos branch that will hopefully get into master some
time after L-2. If interested, see:

http://git.openstack.org/cgit/openstack/neutron/tree/neutron/objects?h=f
eature/qos

Once the feature branch is merged into master, I plan to start
converting existing resources to objects. It may take time and will
definitely span to M. Depending on progress in this regard, we'll see
whether we will be able to consider live data migration. At the
moment, I don't see it happening in M, at least not at the start of it.

> 
> Are there use-cases written down when talking about Neutron
> upgrades?

One thing that is currently in review are partial upgrades. They are
tested for nova (including nova-network) but not for neutron, so it's
considered a nova-network/neutron parity issue.

You should find most of relevant patches in:

https://review.openstack.org/#/q/owner:%22Russell+Bryant%22+status:open,
n,z

Ihar
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQEcBAEBCAAGBQJVnoDyAAoJEC5aWaUY1u57I0QH/AoRSQhBy2m0A1oZ3YtRH/At
2tIwDp83uYa5B3UvxW/NaGxBZW0kra0Wi+kYFv4bv5Uyl2PwMjixkWptPVr/55oc
osVDl1/+r+NQEQeW94XOp0DEy7yekqPqnYi3O72Icggyrj5+l3CjQXkXrXWXzKFh
hwnKMS8kirCrkciqoQCRkAsDDHGHvqC36xHNdDEJXz0OMzmosvNgvpfQOShjfxrX
GRNmah1lKYXkTzpyuhMkdRuCxybDkzpN2rmIbaPziGb5OnOQ3f889H0VsAYh77eJ
vJlIIcTKWiW3qOSs1Pqzmu2PBRewLPA2hF6VshIo7pK4B3TOe0oBKG7dSfepYlQ=
=hbe1
-----END PGP SIGNATURE-----



More information about the OpenStack-dev mailing list