Open Stack

On 05/17/2016 06:30 PM, Crag Wolfe wrote:
> Hi all,
>
> I've read that versioned objects are favored for supporting different
> versions between RPC services and to support rolling upgrades. I'm
> looking to follow the pattern for Heat. Basically, it is the classic
> problem where we want to migrate from writing to a column in one table
> to having that column live in a different table. Looking at nova code,
> the version for a given versioned object is a constant in the given
> object/<the_object_name>.py file. To properly support rolling upgrades
> where we have older and newer heat-engine processes running
> simultaneously (thus avoiding downtime), we have to write to both the
> old column and the new column. Once all processes have been upgraded,
> we can upgrade again to only write to the new location (but still able
> to read from the old location of course). Following the existing
> pattern, this means the operator has to upgrade <the_object_name.py>
> twice (it may be possible to increment VERSION in <the_object_name.py>
> only once, however, the first time).
>
> The drawback of the above is it means cutting two releases (since two
> different .py files). However, I wanted to check if anyone has gone
> with a different approach so only one release is required. One way to
> do that would be by specifying a version (or some other flag) in
> heat.conf. Then, only one <the_object_name>.py release would be
> required -- the logic of whether to write to both the old and new
> location (the intermediate step) versus just the new location (the
> final step) would be in <the_object_name>.py, dictated by the config
> value. The advantage to this approach is now there is only one .py
> file released, though the operator would still have to make a config
> change and restart heat processes a second time to move from the
> intermediate step to the final step.

Nova has the pattern of being able to do all that in one release by
exercising o.vo, but there are assumptions they are relying on (details
[1]):

  * nova-compute accesses the DB through nova-conductor.
  * nova-conductor gets upgraded atomically.
  * nova-conductor is able to backport an object if nova-compute is
    older and doesn't understand it.

Now if you want to have heat-engines running in different versions and
all of them are freely accessing the DB, then that approach won't work
as there's no one who can do a backport.

We've faced same issue in Cinder and developed a way to do such
modifications in three releases for columns that are writable and two
releases for columns that are read-only. This is explained in spec [2]
and devref [3]. And yes, it's a little painful.

If I got everything correctly, your idea of two-step upgrade will work
only for read-only columns. Consider this situation:

 1. We have deployment running h-eng (A and B) in version X.
 2. We apply X+1 migration moving column `foo` to `bar`.
 3. We upgrade h-eng A to X+1. Now it writes to both `foo` and `bar`.
 4. A updates `foo` and `bar`.
 5. B updates `foo`. Now correct value is in `foo` only.
 6. A want to read the value. But is latest one in `foo` or `bar`? No
    way to tell that.


I know Keystone team is trying to solve that with some SQLAlchemy magic,
but I don't think the design is agreed on yet. There was a presentation
at the summit [4] that mentions it (and attempts clarification of
approaches taken by different projects).

Hopefully this helps a little.

Thanks,
Michal (dulek on freenode)

[1] http://www.danplanet.com/blog/2015/10/07/upgrades-in-nova-database-migrations/

[2] http://specs.openstack.org/openstack/cinder-specs/specs/mitaka/online-schema-upgrades.html

[3] http://docs.openstack.org/developer/cinder/devref/rolling.upgrades.html#database-schema-and-data-migrations

[4] https://www.youtube.com/watch?v=ivcNI7EHyAY