<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<div class="moz-cite-prefix">On 05.02.2016 12:28, Salvatore Orlando
wrote:<br>
</div>
<blockquote
cite="mid:CAP0B2WN86gzBPEmRXAOTja4+9hjmEHYBiLnKuDo4t1zEOM4zXg@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On 5 February 2016 at 04:12, Armando
M. <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:armamig@gmail.com" target="_blank">armamig@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">
<div>
<div class="h5">On 4 February 2016 at 08:22, John
Belamaric <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:jbelamaric@infoblox.com"
target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:jbelamaric@infoblox.com">jbelamaric@infoblox.com</a></a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex"><span><br>
> On Feb 4, 2016, at 11:09 AM, Carl
Baldwin <<a moz-do-not-send="true"
href="mailto:carl@ecbaldwin.net"
target="_blank">carl@ecbaldwin.net</a>>
wrote:<br>
><br>
> On Thu, Feb 4, 2016 at 7:23 AM, Pavel
Bondar <<a moz-do-not-send="true"
href="mailto:pbondar@infoblox.com"
target="_blank">pbondar@infoblox.com</a>>
wrote:<br>
>> I am trying to bring more attention
to [1] to make final decision on<br>
>> approach to use.<br>
>> There are a few point that are not
100% clear for me at this point.<br>
>><br>
>> 1) Do we plan to switch all current
clouds to pluggable ipam<br>
>> implementation in Mitaka?<br>
><br>
> I think our plan originally was only to
deprecate the non-pluggable<br>
> implementation in Mitaka and remove it
in Newton. However, this is<br>
> worth some more consideration. The
pluggable version of the reference<br>
> implementation should, in theory, be at
parity with the current<br>
> non-pluggable implementation. We've
tested it before and shown<br>
> parity. What we're missing is regular
testing in the gate to ensure<br>
> it continues this way.<br>
><br>
<br>
</span>Yes, it certainly should be at parity,
and gate testing to ensure it would be best.<br>
<span><br>
>> yes --><br>
>> Then data migration can be done as
alembic_migration and it is what<br>
>> currently implemented in [2] PS54.<br>
>> In this case during upgrade from
Liberty to Mitaka all users are<br>
>> unconditionally switched to
reference ipam driver<br>
>> from built-in ipam implementation.<br>
>> If operator wants to continue using
build-in ipam implementation it can<br>
>> manually turn off ipam_driver in
neutron.conf<br>
>> immediately after upgrade (data is
not deleted from old tables).<br>
><br>
> This has a certain appeal to it. I
think the migration will be<br>
> straight-forward since the table
structure doesn't really change much.<br>
> Doing this as an alembic migration
would be the easiest from an<br>
> upgrade point of view because it fits
seamlessly in to our current<br>
> upgrade strategy.<br>
><br>
> If we go this way, we should get this
in soon so that we can get the<br>
> gate and others running with this code
for the remainder of the cycle.<br>
><br>
<br>
</span>If we do this, and the operator reverts
back to the non-pluggable version,<br>
then we will leave stale records in the new
IPAM tables. At the very least,<br>
we would need a way to clean those up and to
migrate at a later time.<br>
<span><br>
>> no --><br>
>> Operator is free to choose whether
it will switch to pluggable ipam<br>
>> implementation<br>
>> and when. And it leads to no
automatic data migration.<br>
>> In this case operator is supplied
with script for migration to pluggable<br>
>> ipam (and probably from pluggable
ipam),<br>
>> which can be executed by operator
during upgrade or at any point after<br>
>> upgrade is done.<br>
>> I was testing this approach in [2]
PS53 (have unresolved issues in it<br>
>> for now).<br>
><br>
> If there is some risk in changing over
then this should still be<br>
> considered. But, the more I think
about it, the more I think that we<br>
> should just make the switch seamlessly
for the operator and be done<br>
> with it. This approach puts a certain
burden on the operator to<br>
> choose when to do the migration and go
through the steps manually to<br>
> do it. And, since our intention is to
deprecate and remove the<br>
> non-pluggable implementation, it is
inevitable that they will have to<br>
> eventually switch anyway.<br>
><br>
> This also makes testing much more
difficult. If we go this route, we<br>
> really should be testing both equally.
Does this mean that we need to<br>
> set up a whole new job to run the
pluggable implementation along side<br>
> the old implementation? This kind of
feels like a nightmare to me.<br>
> What do you think?<br>
><br>
<br>
</span>Originally (as I mentioned in the
meeting), I was thinking that we should not
automatically migrate. However, I see the
appeal of your arguments. Seamless is best, of
course. But if we offer going back to
non-pluggable, (which I think we need to at
this point in the Mitaka cycle), we probably
need to provide a script as mentioned above.
Seems feasible, though.<br>
<div>
<div><br>
<br>
<br>
</div>
</div>
</blockquote>
<div><br>
</div>
</div>
</div>
<div>We're tackling more than one issue in this
thread and I am having a hard time wrapping my
head around it. Let me try to sum it all up.</div>
<div><br>
</div>
<div>a) switching from non-pluggable to pluggable
it's a matter of running a data migration + a
config change</div>
<div>b) We can either switch automatically on
restart (option b1) or manually on operator
command (b2)</div>
<div>c) Do we make pluggable ipam default and when</div>
<div>d) Testing the migration</div>
<div>e) Deprecating the non-pluggable one.<br>
</div>
<div><br>
</div>
<div>I hope we are all in agreement on bullet point
a), because knowing the complexity of your problem
is halfway to our solution.<br>
</div>
<div><br>
</div>
<div>as for b) I think that manual migration is best
for two reasons: 1) In HA scenarios, seamless
upgrade (ie. on server restart) can be a
challenge; 2) the operator must 'manually' change
the driver, so he/she is very conscious of what
he/she is doing and can take enough precautions
should something go astray. Technically we can
make this as sophisticated and seamless as we
want, but this is a one-off, once it's done the
pain goes away, and we won't be doing another
migration like this ever again. So I wouldn't over
engineer it.</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Agreed. Operators love to automate things, but they
generally don't like when components automatically do
things they maybe do not expect to do (I don't think we
should assume all operators fully read release notes). So
the manual step is preferable, and not that painful after
all. From an historical perspective, a manual switch was
the same approach adopted for migration from OVS/LB
plugins to ML2.</div>
<div> </div>
</div>
</div>
</div>
</blockquote>
That is fine for me, so I'll focus on script approach where operator
manually changes driver and run migrate script.<br>
<blockquote
cite="mid:CAP0B2WN86gzBPEmRXAOTja4+9hjmEHYBiLnKuDo4t1zEOM4zXg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>as for c) I think it's a little late to make
pluggable ipam default in Mitaka; I'd rather
switch defaults early in the cycle (depending on
the entity of the config) and this one seems
serious enough that I'd rather have enough
exercising in the gate to prove it solid. In a
nutshell: let's defer the driver switch to N. When
we do, we'll have to worry about grenade, but
Grenade can run scripts and we can 'emulate' the
operator hand.</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>While this should be low risk, doing such a change in
the 3rd milestone really means testing your luck. I think
the core team is already busy with a massive backlog of
blueprints and they really could without chasing issues in
the reference IPAM driver.</div>
<div> </div>
</div>
</div>
</div>
</blockquote>
Agree, switching driver early in N sounds like the safest option.<br>
<blockquote
cite="mid:CAP0B2WN86gzBPEmRXAOTja4+9hjmEHYBiLnKuDo4t1zEOM4zXg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>as for d), and in preparation for the default
switch, I think we can come up with an
experimental (or periodic) grenade 'side-way' job
where we validate only the ipam driver switch.
It's best to do this on a recurring basis rather
than on a continuous basis. </div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Forgive, my ignorance, but do we also have a job,
perhaps in the experimental queue, that runs api,
full-stack, functional, and tempest job on Neutron running
with the IPAM driver. If not we might want to have that in
the experimental queue.</div>
<div>As for migration testjng, the periodic job will be good
enough, but it won't harm adding to to the experimental
queue as well.</div>
</div>
</div>
</div>
</blockquote>
Currently IPAM Driver is tested by UTs and only a few functional
tests are tuned to utilize pluggable ipam code,<br>
so we don't have dedicated api, full-stack, functional, and tempest
jobs on Neutron running with the IPAM driver.<br>
And I would vote for having them, or at least some of them as
periodic/experimental jobs.<br>
I am not aware of workflow for adding new jobs for jenkins, so it
would be nice if someone could drive this process or<br>
provide information on how I could do it.<br>
<br>
As I got current plan to merge migration to pluggable ipam before
mitaka-3,<br>
so what kind of tests should be included to make it happen?<br>
As Carl mentioned, we may want to have grenade jobs to verify
migration during upgrade and after upgrade,<br>
so would like to clarify that.<br>
<br>
Thanks,<br>
Pavel<br>
<br>
<blockquote
cite="mid:CAP0B2WN86gzBPEmRXAOTja4+9hjmEHYBiLnKuDo4t1zEOM4zXg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>as for e) I think we cannot afford to deprecate
the non-pluggable one in back-to-back cycles, but
probably we'll have to stretch a little longer
once we have enough field feedback (via user
survey) that the switch is well under way. Rather
than forcing the upgrade to the operators, let's
hear from them that they have embraced the new
IPAM module. If things go slow we can nudge them
via evangelism :) I believe this is the only way
to provide the smoothest and least painful
experience. We can afford to keep some debt
around, we freed ourselves of lots of code in the
last cycle or two!</div>
</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>There is no additional debt in my opinion, It's just
the normal evolution of the software. We've introduced a
new engine and we have a path to switch from the old to
the new, and now we can start the deprecation process. I
think it's a reasonable goal to deprecate by Ocata release
and then kill the old IPAM engine for the P release.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Salvatore</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>HTH</div>
<span class="HOEnZb"><font color="#888888">
<div>Armando </div>
</font></span><span class="">
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0
0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div>
<div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for
usage questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</div>
</blockquote>
</span></div>
<br>
</div>
</div>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage
questions)<br>
Unsubscribe: <a moz-do-not-send="true"
href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe"
rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev-request@lists.openstack.org?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>