Open Stack

Hi all,

This is a report on upgrades related topics discussed during PTG in
Atlanta. A general PTG report from PTL can be found at:

http://lists.openstack.org/pipermail/openstack-dev/2017-February/113032.html

Topics discussed:
    1. OVO progress;
    2. running mixed server versions;
    3. online data migration framework;
    4. impact of new features (multiple port bindings) for mixed
server execution, and how to deal with it;
    5. gate.

=> OVO progress
The effort is a bit stalled but slowly progressing. Progress is
sometimes blocked by external ongoing initiatives: new oslo.db
enginefacade; other code refactoring. Sometimes we need to back off
and fix plugin database logic before switching to objects (think of
lock_for_update removal in all places in current code). The initiative
is critical only as long as there are other initiatives that would
benefit from it. At this point, we are mainly looking at multiple port
bindings feature and its potential impact on our ability to run mixed
server versions at the same time. Other objects are worth an effort
but are not critical. There may be other objects during the cycle that
may be determined critical for rolling upgrades support.

Specifically talking about lock_for_update support in objects layer:
https://review.openstack.org/#/c/435598/ , Kevin is hesitant to add
it. Instead we are going to clean up remaining code that uses the
locking feature, like in https://review.openstack.org/#/c/438144/ . We
need the same approach applied for quotas.

=> Running mixed server versions
Initial testing done by Artur showed that it's now possible to execute
Newton and Ocata server versions in the same environment with some
success. This is partly the result of OVO work, but also the fact that
Ocata cycle was short and not rich on new features. We should build on
top of that to deliver the same for Pike. We should give special
attention to initiatives that may disrupt the work (multiple port
bindings and others).

=> Online data migration framework
We discussed that in general there are 3 steps needed for online
migration 1) Declaration of the intent and creating a general
framework for online migration, 2) CLI for online migration (example
patch https://review.openstack.org/#/c/432494/) and, 3) The changes
made in the object layer for online data migration needs to be
backward compatible. We sat down and looked at specific contract
migrations from the past as a matter of case study excercise. During
our discussion we found out that all the migrations might not be
addressed by the general framework and will require case-by-case code
to implement the needed transition. Actually, intent for most
non-obvious contractions would be hard to express in a general way.

=> Impact of multiple port bindings
Discussions suggest that the feature may be of high impact for our
ability to deliver mixed server versions for Ocata->Pike upgrade. This
is not because of database access not being properly aligned for the
newly expected feature. Instead, we may hit issues because of the
nature of the extension usage, that is usually consumed by compute
component. Once nova switches to using the new extension to drive port
bindings if the extension is present, and if neutron replies to nova
that it indeed supports the new extension before all neutron-servers
are upgraded to Pike, then it may turn out that some data stored in
the database may not be easily implemented/acted on by older
neutron-servers.

Ideally, we would instead block any new API requests till the moment
when all neutron-servers are running the new code, at which point they
could start advertising the fact on /extensions/ endpoint. That would
imply that neutron-servers become aware of each other, and extensions
each of them support and expose. In that case, they could negotiate to
use the common subset of extensions, and avoid exposing any extensions
that are not implemented by any other neutron-server. In that case,
newer servers would still behave as if they don't support new API
features. Then once the upgrade to the new release is complete,
neutron-servers could detect the end of upgrade phase (either through
some external event, like admin initiated signals; or by periodic
inspection of the server db table) and reload extensions to enable new
API. I am going to report the proposal as RFE and write it up in form
of a spec where we will be able to discuss it in more details.

Since multiple port bindings work is scheduled to land this cycle, we
should work on that proposal in Pike too.

=> Gating
It's still not clear what to do with mixed server version gating, and
apparently we will need to sync again with folks driving it for nova
gate. In the meantime, we may want to continue periodic local
validations of mixed setup to assure nothing is borked.

As for grenade multinode linuxbridge job, there is lack of clear
interest in the subgroup, so I asked Kevin if he is interested in
making it working. Kevin said he will take a look and take it from
where it is right now.

Ihar