Open Stack

On 05.02.2016 12:28, Salvatore Orlando wrote:
>
>
> On 5 February 2016 at 04:12, Armando M. <armamig at gmail.com
> <mailto:armamig at gmail.com>> wrote:
>
>
>
>     On 4 February 2016 at 08:22, John Belamaric
>     <jbelamaric at infoblox.com <mailto:jbelamaric at infoblox.com>> wrote:
>
>
>         > On Feb 4, 2016, at 11:09 AM, Carl Baldwin
>         <carl at ecbaldwin.net <mailto:carl at ecbaldwin.net>> wrote:
>         >
>         > On Thu, Feb 4, 2016 at 7:23 AM, Pavel Bondar
>         <pbondar at infoblox.com <mailto:pbondar at infoblox.com>> wrote:
>         >> I am trying to bring more attention to [1] to make final
>         decision on
>         >> approach to use.
>         >> There are a few point that are not 100% clear for me at
>         this point.
>         >>
>         >> 1) Do we plan to switch all current clouds to pluggable ipam
>         >> implementation in Mitaka?
>         >
>         > I think our plan originally was only to deprecate the
>         non-pluggable
>         > implementation in Mitaka and remove it in Newton.  However,
>         this is
>         > worth some more consideration.  The pluggable version of the
>         reference
>         > implementation should, in theory, be at parity with the current
>         > non-pluggable implementation.  We've tested it before and shown
>         > parity.  What we're missing is regular testing in the gate
>         to ensure
>         > it continues this way.
>         >
>
>         Yes, it certainly should be at parity, and gate testing to
>         ensure it would be best.
>
>         >> yes -->
>         >> Then data migration can be done as alembic_migration and it
>         is what
>         >> currently implemented in [2] PS54.
>         >> In this case during upgrade from Liberty to Mitaka all
>         users are
>         >> unconditionally switched to reference ipam driver
>         >> from built-in ipam implementation.
>         >> If operator wants to continue using build-in ipam
>         implementation it can
>         >> manually turn off ipam_driver in neutron.conf
>         >> immediately after upgrade (data is not deleted from old
>         tables).
>         >
>         > This has a certain appeal to it.  I think the migration will be
>         > straight-forward since the table structure doesn't really
>         change much.
>         > Doing this as an alembic migration would be the easiest from an
>         > upgrade point of view because it fits seamlessly in to our
>         current
>         > upgrade strategy.
>         >
>         > If we go this way, we should get this in soon so that we can
>         get the
>         > gate and others running with this code for the remainder of
>         the cycle.
>         >
>
>         If we do this, and the operator reverts back to the
>         non-pluggable version,
>         then we will leave stale records in the new IPAM tables. At
>         the very least,
>         we would need a way to clean those up and to migrate at a
>         later time.
>
>         >> no -->
>         >> Operator is free to choose whether it will switch to
>         pluggable ipam
>         >> implementation
>         >> and when. And it leads to no automatic data migration.
>         >> In this case operator is supplied with script for migration
>         to pluggable
>         >> ipam (and probably from pluggable ipam),
>         >> which can be executed by operator during upgrade or at any
>         point after
>         >> upgrade is done.
>         >> I was testing this approach in [2] PS53 (have unresolved
>         issues in it
>         >> for now).
>         >
>         > If there is some risk in changing over then this should still be
>         > considered.  But, the more I think about it, the more I
>         think that we
>         > should just make the switch seamlessly for the operator and
>         be done
>         > with it.  This approach puts a certain burden on the operator to
>         > choose when to do the migration and go through the steps
>         manually to
>         > do it.  And, since our intention is to deprecate and remove the
>         > non-pluggable implementation, it is inevitable that they
>         will have to
>         > eventually switch anyway.
>         >
>         > This also makes testing much more difficult.  If we go this
>         route, we
>         > really should be testing both equally.  Does this mean that
>         we need to
>         > set up a whole new job to run the pluggable implementation
>         along side
>         > the old implementation?  This kind of feels like a nightmare
>         to me.
>         > What do you think?
>         >
>
>         Originally (as I mentioned in the meeting), I was thinking
>         that we should not automatically migrate. However, I see the
>         appeal of your arguments. Seamless is best, of course. But if
>         we offer going back to non-pluggable, (which I think we need
>         to at this point in the Mitaka cycle), we probably need to
>         provide a script as mentioned above. Seems feasible, though.
>
>
>
>
>     We're tackling more than one issue in this thread and I am having
>     a hard time wrapping my head around it. Let me try to sum it all up.
>
>     a) switching from non-pluggable to pluggable it's a matter of
>     running a data migration + a config change
>     b) We can either switch automatically on restart (option b1) or
>     manually on operator command (b2)
>     c) Do we make pluggable ipam default and when
>     d) Testing the migration
>     e) Deprecating the non-pluggable one.
>
>     I hope we are all in agreement on bullet point a), because knowing
>     the complexity of your problem is halfway to our solution.
>
>     as for b) I think that manual migration is best for two reasons:
>     1) In HA scenarios, seamless upgrade (ie. on server restart) can
>     be a challenge; 2) the operator must 'manually' change the driver,
>     so he/she is very conscious of what he/she is doing and can take
>     enough precautions should something go astray. Technically we can
>     make this as sophisticated and seamless as we want, but this is a
>     one-off, once it's done the pain goes away, and we won't be doing
>     another migration like this ever again. So I wouldn't over
>     engineer it.
>
>
> Agreed. Operators love to automate things, but they generally don't
> like when components automatically do things they maybe do not expect
> to do (I don't think we should assume all operators fully read release
> notes). So the manual step is preferable, and not that painful after
> all. From an historical perspective, a manual switch was the same
> approach adopted for migration from OVS/LB plugins to ML2.
>  
That is fine for me, so I'll focus on script approach where operator
manually changes driver and run migrate script.
>
>
>     as for c) I think it's a little late to make pluggable ipam
>     default in Mitaka; I'd rather switch defaults early in the cycle
>     (depending on the entity of the config) and this one seems serious
>     enough that I'd rather have enough exercising in the gate to prove
>     it solid. In a nutshell: let's defer the driver switch to N. When
>     we do, we'll have to worry about grenade, but Grenade can run
>     scripts and we can 'emulate' the operator hand.
>
>
> While this should be low risk, doing such a change in the 3rd
> milestone really means testing your luck. I think the core team is
> already busy with a massive backlog of blueprints and they really
> could without chasing issues in the reference IPAM driver.
>  
Agree, switching driver early in N sounds like the safest option.
>
>
>     as for d), and in preparation for the default switch, I think we
>     can come up with an experimental (or periodic) grenade 'side-way'
>     job where we validate only the ipam driver switch. It's best to do
>     this on a recurring basis rather than on a continuous basis. 
>
>
> Forgive, my ignorance, but do we also have a job, perhaps in the
> experimental queue, that runs api, full-stack, functional, and tempest
> job on Neutron running with the IPAM driver. If not we might want to
> have that in the experimental queue.
> As for migration testjng, the periodic job will be good enough, but it
> won't harm adding to to the experimental queue as well.
Currently IPAM Driver is tested by UTs and only a few functional tests
are tuned to utilize pluggable ipam code,
so we don't have dedicated api, full-stack, functional, and tempest jobs
on Neutron running with the IPAM driver.
And I would vote for having them, or at least some of them as
periodic/experimental jobs.
I am not aware of workflow for adding new jobs for jenkins, so it would
be nice if someone could drive this process or
provide information on how I could do it.

As I got current plan to merge migration to pluggable ipam before mitaka-3,
so what kind of tests should be included to make it happen?
As Carl mentioned, we may want to have grenade jobs to verify migration
during upgrade and after upgrade,
so would like to clarify that.

Thanks,
Pavel

>     as for e) I think we cannot afford to deprecate the non-pluggable
>     one in back-to-back cycles, but probably we'll have to stretch a
>     little longer once we have enough field feedback (via user survey)
>     that the switch is well under way. Rather than forcing the upgrade
>     to the operators, let's hear from them that they have embraced the
>     new IPAM module. If things go slow we can nudge them via
>     evangelism :) I believe this is the only way to provide the
>     smoothest and least painful experience. We can afford to keep some
>     debt around, we freed ourselves of lots of code in the last cycle
>     or two!
>
>
> There is no additional debt in my opinion, It's just the normal
> evolution of the software. We've introduced a new engine and we have a
> path to switch from the old to the new, and now we can start the
> deprecation process. I think it's a reasonable goal to deprecate by
> Ocata release and then kill the old IPAM engine for the P release.
>
> Cheers,
> Salvatore
>  
>
>
>     HTH
>     Armando 
>
>      
>
>         __________________________________________________________________________
>         OpenStack Development Mailing List (not for usage questions)
>         Unsubscribe:
>         OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>         <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160205/a35ce199/attachment.html>