Open Stack

On 07/09/2015 10:11 AM, Ihar Hrachyshka wrote:
> On 07/09/2015 09:01 AM, Artur Korzen wrote:
>> Hi all,
> 
>> I've been researching the Neutron project as a part of work on 
>> Openstack rolling upgrades, my primary assignments included
>> testing if there is no VM access downtime when performing
>> upgrade.
> 
> 
> 
>> Are you aware of any issues that are present in Liberty release
>> of Neutron, that may cause the VM access downtime and may stop
>> ops to pick up newest versions of code?
> 
> 
> 
>> When talking about no VM access downtime during upgrade, the 
>> sensitive operations are:
> 
>> 1. ensure that even after neutron services is
>> shutoff/uninstalled, the traffic can go into the VM 2. take care
>> on neutron services startup/installation that existing
>> configuration is preserved (routing tables, forwarding entries
>> and security rules) 3. the underlying network technologies (OVS,
>> linux bridges etc.) is working without distraction when upgrading
>> neutron code: no dropping table flows or applying the traffic
>> rules to drop the packages
> 
> 
> 
>> To be compatible between different versions of service talking
>> to each other, the important areas are:
> 
>> 1. RPC API versioning 2. Objects exchanged between services via 
>> RPC 3. Database schema (bp [2]) and data migration
> 
> 
> 
>> I have researched the ML2 plugin with ovs agent, including the 
>> neutron-ovs-agent, dhcp-server, l3-agent, neutron-server.
> 
>> One of the VM access downtime during upgrade is addressed in [1],
>>  removing the flows in ovs after neutron agent restarts.
> 
> 
> 
>> Is Neutron ready to be upgradable with minimal downtime of
>> services and no VM access downtime?
> 
> As the ovs bug you refer to above, no, at least not in reference 
> implementation. That's for data plane.
> 
> As for other services, neutron-server online schema migration
> should help, and I hope to get it implemented in L (though there
> are some obstacles that may block us; we'll see).
> 
> As for live data migration, neutron code base is not ready yet to 
> reasonably require live data migration being implemented in all 
> patches that need data moves in database. The very first obstacle
> for that is that there is no middleware layer between sqlalchemy
> and the rest of neutron that would allow us to hide migration
> details. Oslo.versionedobjects is such a middleware. See below.
> 
> 
>> Are there any guidelines on using the oslo versionobjects and its
>>  priority in Liberty cycle?
> 
> It's not a common priority for L, but we've started on this road 
> inside feature/qos branch that will hopefully get into master some 
> time after L-2. If interested, see:
> 
> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/objects?h=f
>
> 
eature/qos
> 
> Once the feature branch is merged into master, I plan to start 
> converting existing resources to objects. It may take time and
> will definitely span to M. Depending on progress in this regard,
> we'll see whether we will be able to consider live data migration.
> At the moment, I don't see it happening in M, at least not at the
> start of it.
> 
> 
>> Are there use-cases written down when talking about Neutron 
>> upgrades?
> 
> One thing that is currently in review are partial upgrades. They
> are tested for nova (including nova-network) but not for neutron,
> so it's considered a nova-network/neutron parity issue.
> 
> You should find most of relevant patches in:
> 
> https://review.openstack.org/#/q/owner:%22Russell+Bryant%22+status:open,
>
> 
n,z

Yes, there's a ton of work to make rolling upgrades as robust as what
Nova has done.  There's significant limitations to what Neutron can do
without breaking it, but hopefully the grenade job would help us catch
things that would break it sooner.  At the moment those jobs have been
blocked pending some re-work of how the partial upgrade jobs work.

As of the last release (Kilo), rolling upgrades from Juno with Neutron
worked when verified manually.  I'm hoping we'll have the same for
Liberty, but I'm not sure if anyone has tried it out recently.

All of the things discussed here sound like good things to work on.  I
actually thought a group was going to work on versioned objects for
Kilo, but that never materialized.

I'll also plug a project I'm working on (OVN, part of the Open vSwitch
project), which I also think simplifies this a good bit for Neutron,
because you won't have to run Neutron agents on compute nodes anymore.

-- 
Russell Bryant