Open Stack

+1 on avoiding changes that break rolling upgrade.

Rolling upgrade has been working so far (at least from my perspective), and
as openstack adoption spreads, it will be important for more and more users.

How do we make rolling upgrade a "supported" part of Neutron?

- Jack

> -----Original Message-----
> From: Assaf Muller [mailto:amuller at redhat.com]
> Sent: Thursday, March 05, 2015 11:59 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [Neutron] Issue when upgrading from Juno to Kilo due
> to agent report_state RPC namespace patch
> 
> 
> 
> ----- Original Message -----
> > To turn this stuff off, you don't need to revert.  I'd suggest just
> > setting the namespace contants to None, and that will result in the same
> > thing.
> >
> >
> http://git.openstack.org/cgit/openstack/neutron/tree/neutron/common/constants.py#
> n152
> >
> > It's definitely a non-backwards compatible change.  That was a conscious
> > choice as the interfaces are a bit of a tangled mess, IMO.  The
> > non-backwards compatible changes were simpler so I went that route,
> > because as far as I could tell, rolling upgrades were not supported.  If
> > they do work, it's due to luck.  There's multiple things including the
> > lack of testing this scenario to lack of data versioning that make it a
> > pretty shaky area.
> >
> > However, if it worked for some people, I totally get the argument
> > against breaking it intentionally.  As mentioned before, a quick fix if
> > needed is to just set the namespace constants to None.  If someone wants
> > to do something to make it backwards compatible, that's even better.
> >
> 
> I sent out an email to the operators list to get some feedback:
> http://lists.openstack.org/pipermail/openstack-operators/2015-March/006429.html
> 
> And at least one operator reported that he performed a rolling Neutron upgrade
> from I to J successfully. So, I'm agreeing with you agreeing with me that we
> probably don't want to mess this up knowingly, even though there is no testing
> to make sure that it keeps working.
> 
> I'll follow up on IRC with you to figure out who's doing what.
> 
> > --
> > Russell Bryant
> >
> > On 03/04/2015 11:50 AM, Salvatore Orlando wrote:
> > > To put in another way I think we might say that change 154670 broke
> > > backward compatibility on the RPC interface.
> > > To be fair this probably happened because RPC interfaces were organised
> > > in a way such that this kind of breakage was unavoidable.
> > >
> > > I think the strategy proposed by Assaf is a viable one. The point about
> > > being able to do rolling upgrades only from version N to N+1 is a
> > > sensible one, but it has more to do with general backward compability
> > > rules for RPC interfaces.
> > >
> > > In the meanwhile this is breaking a typical upgrade scenario. If a fix
> > > allowing agent state updates both namespaced and not is available today
> > > or tomorrow, that's fine. Otherwise I'd revert just to be safe.
> > >
> > > By the way, we were supposed to have already removed all server rpc
> > > callbacks in the appropriate package... did we forget out this one or is
> > > there a reason for which it's still in neutron.db?
> > >
> > > Salvatore
> > >
> > > On 4 March 2015 at 17:23, Miguel Ángel Ajo <majopela at redhat.com
> > > <mailto:majopela at redhat.com>> wrote:
> > >
> > >     I agree with Assaf, this is an issue across updates, and
> > >     we may want (if that’s technically possible) to provide
> > >     access to those functions with/without namespace.
> > >
> > >     Or otherwise think about reverting for now until we find a
> > >     migration strategy
> > >
> > >
> https://review.openstack.org/#/q/status:merged+project:openstack/neutron+branch:m
> aster+topic:bp/rpc-docs-and-namespaces,n,z
> > >
> > >
> > >     Best regards,
> > >     Miguel Ángel Ajo
> > >
> > >     On Wednesday, 4 de March de 2015 at 17:00, Assaf Muller wrote:
> > >
> > >>     Hello everyone,
> > >>
> > >>     I'd like to highlight an issue with:
> > >>     https://review.openstack.org/#/c/154670/
> > >>
> > >>     According to my understanding, most deployments upgrade the
> > >>     controllers first
> > >>     and compute/network nodes later. During that time period, all
> > >>     agents will
> > >>     fail to report state as they're sending the report_state message
> > >>     outside
> > >>     of any namespace while the server is expecting that message in a
> > >>     namespace.
> > >>     This is a show stopper as the Neutron server will think all of its
> > >>     agents are dead.
> > >>
> > >>     I think the obvious solution is to modify the Neutron server code
> > >>     so that
> > >>     it accepts the report_state method both in and outside of the
> > >>     report_state
> > >>     RPC namespace and chuck that code away in L (Assuming we support
> > >>     rolling upgrades
> > >>     only from version N to N+1, which while is unfortunate, is the
> > >>     behavior I've
> > >>     seen in multiple places in the code).
> > >>
> > >>     Finally, are there additional similar issues for other RPC methods
> > >>     placed in a namespace
> > >>     this cycle?
> > >>
> > >>
> > >>     Assaf Muller, Cloud Networking Engineer
> > >>     Red Hat
> > >>