Open Stack

On 5 February 2016 at 04:12, Armando M. <armamig at gmail.com> wrote:

>
>
> On 4 February 2016 at 08:22, John Belamaric <jbelamaric at infoblox.com>
> wrote:
>
>>
>> > On Feb 4, 2016, at 11:09 AM, Carl Baldwin <carl at ecbaldwin.net> wrote:
>> >
>> > On Thu, Feb 4, 2016 at 7:23 AM, Pavel Bondar <pbondar at infoblox.com>
>> wrote:
>> >> I am trying to bring more attention to [1] to make final decision on
>> >> approach to use.
>> >> There are a few point that are not 100% clear for me at this point.
>> >>
>> >> 1) Do we plan to switch all current clouds to pluggable ipam
>> >> implementation in Mitaka?
>> >
>> > I think our plan originally was only to deprecate the non-pluggable
>> > implementation in Mitaka and remove it in Newton.  However, this is
>> > worth some more consideration.  The pluggable version of the reference
>> > implementation should, in theory, be at parity with the current
>> > non-pluggable implementation.  We've tested it before and shown
>> > parity.  What we're missing is regular testing in the gate to ensure
>> > it continues this way.
>> >
>>
>> Yes, it certainly should be at parity, and gate testing to ensure it
>> would be best.
>>
>> >> yes -->
>> >> Then data migration can be done as alembic_migration and it is what
>> >> currently implemented in [2] PS54.
>> >> In this case during upgrade from Liberty to Mitaka all users are
>> >> unconditionally switched to reference ipam driver
>> >> from built-in ipam implementation.
>> >> If operator wants to continue using build-in ipam implementation it can
>> >> manually turn off ipam_driver in neutron.conf
>> >> immediately after upgrade (data is not deleted from old tables).
>> >
>> > This has a certain appeal to it.  I think the migration will be
>> > straight-forward since the table structure doesn't really change much.
>> > Doing this as an alembic migration would be the easiest from an
>> > upgrade point of view because it fits seamlessly in to our current
>> > upgrade strategy.
>> >
>> > If we go this way, we should get this in soon so that we can get the
>> > gate and others running with this code for the remainder of the cycle.
>> >
>>
>> If we do this, and the operator reverts back to the non-pluggable version,
>> then we will leave stale records in the new IPAM tables. At the very
>> least,
>> we would need a way to clean those up and to migrate at a later time.
>>
>> >> no -->
>> >> Operator is free to choose whether it will switch to pluggable ipam
>> >> implementation
>> >> and when. And it leads to no automatic data migration.
>> >> In this case operator is supplied with script for migration to
>> pluggable
>> >> ipam (and probably from pluggable ipam),
>> >> which can be executed by operator during upgrade or at any point after
>> >> upgrade is done.
>> >> I was testing this approach in [2] PS53 (have unresolved issues in it
>> >> for now).
>> >
>> > If there is some risk in changing over then this should still be
>> > considered.  But, the more I think about it, the more I think that we
>> > should just make the switch seamlessly for the operator and be done
>> > with it.  This approach puts a certain burden on the operator to
>> > choose when to do the migration and go through the steps manually to
>> > do it.  And, since our intention is to deprecate and remove the
>> > non-pluggable implementation, it is inevitable that they will have to
>> > eventually switch anyway.
>> >
>> > This also makes testing much more difficult.  If we go this route, we
>> > really should be testing both equally.  Does this mean that we need to
>> > set up a whole new job to run the pluggable implementation along side
>> > the old implementation?  This kind of feels like a nightmare to me.
>> > What do you think?
>> >
>>
>> Originally (as I mentioned in the meeting), I was thinking that we should
>> not automatically migrate. However, I see the appeal of your arguments.
>> Seamless is best, of course. But if we offer going back to non-pluggable,
>> (which I think we need to at this point in the Mitaka cycle), we probably
>> need to provide a script as mentioned above. Seems feasible, though.
>>
>>
>>
>>
> We're tackling more than one issue in this thread and I am having a hard
> time wrapping my head around it. Let me try to sum it all up.
>
> a) switching from non-pluggable to pluggable it's a matter of running a
> data migration + a config change
> b) We can either switch automatically on restart (option b1) or manually
> on operator command (b2)
> c) Do we make pluggable ipam default and when
> d) Testing the migration
> e) Deprecating the non-pluggable one.
>
> I hope we are all in agreement on bullet point a), because knowing the
> complexity of your problem is halfway to our solution.
>
> as for b) I think that manual migration is best for two reasons: 1) In HA
> scenarios, seamless upgrade (ie. on server restart) can be a challenge; 2)
> the operator must 'manually' change the driver, so he/she is very conscious
> of what he/she is doing and can take enough precautions should something go
> astray. Technically we can make this as sophisticated and seamless as we
> want, but this is a one-off, once it's done the pain goes away, and we
> won't be doing another migration like this ever again. So I wouldn't over
> engineer it.
>

Agreed. Operators love to automate things, but they generally don't like
when components automatically do things they maybe do not expect to do (I
don't think we should assume all operators fully read release notes). So
the manual step is preferable, and not that painful after all. From an
historical perspective, a manual switch was the same approach adopted for
migration from OVS/LB plugins to ML2.


>
> as for c) I think it's a little late to make pluggable ipam default in
> Mitaka; I'd rather switch defaults early in the cycle (depending on the
> entity of the config) and this one seems serious enough that I'd rather
> have enough exercising in the gate to prove it solid. In a nutshell: let's
> defer the driver switch to N. When we do, we'll have to worry about
> grenade, but Grenade can run scripts and we can 'emulate' the operator hand.
>

While this should be low risk, doing such a change in the 3rd milestone
really means testing your luck. I think the core team is already busy with
a massive backlog of blueprints and they really could without chasing
issues in the reference IPAM driver.


>
> as for d), and in preparation for the default switch, I think we can come
> up with an experimental (or periodic) grenade 'side-way' job where we
> validate only the ipam driver switch. It's best to do this on a recurring
> basis rather than on a continuous basis.
>

Forgive, my ignorance, but do we also have a job, perhaps in the
experimental queue, that runs api, full-stack, functional, and tempest job
on Neutron running with the IPAM driver. If not we might want to have that
in the experimental queue.
As for migration testjng, the periodic job will be good enough, but it
won't harm adding to to the experimental queue as well.

>
> as for e) I think we cannot afford to deprecate the non-pluggable one in
> back-to-back cycles, but probably we'll have to stretch a little longer
> once we have enough field feedback (via user survey) that the switch is
> well under way. Rather than forcing the upgrade to the operators, let's
> hear from them that they have embraced the new IPAM module. If things go
> slow we can nudge them via evangelism :) I believe this is the only way to
> provide the smoothest and least painful experience. We can afford to keep
> some debt around, we freed ourselves of lots of code in the last cycle or
> two!
>

There is no additional debt in my opinion, It's just the normal evolution
of the software. We've introduced a new engine and we have a path to switch
from the old to the new, and now we can start the deprecation process. I
think it's a reasonable goal to deprecate by Ocata release and then kill
the old IPAM engine for the P release.

Cheers,
Salvatore


>
> HTH
> Armando
>
>
>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160205/dc9a0804/attachment.html>