Open Stack

Hi,

I'd like to discuss how we can have DB upgrade migration method (with 
data manipulation) in placement.

---

BackGround
==========

https://bugs.launchpad.net/nova/+bug/1803925

* In Rocky, to have nested resource provider feature, we expanded the DB 
to have root provider id column.
     * The root provider id shouldn't be None and for root providers it 
should be the same value of its resource provider id.

* In Rocky, the code is build in a backward compatible way doing online 
migration.
     * For each request of listing/showing resource providers, we look 
the root provider id and if it is stale and empty, we assume the 
resource provider is a root and set the same value as resource provider id.
     * Those providers that are not called in the Rocky cycle will still 
have an empty value for the root provider id.

* In Stein or later, we want a way to be sure that all the root provider 
id contains some non-None value.
     * This is because we have a lot of TODOs in code which we want to 
clean up once we are sure all the root provider ids have non-None value 
in the DB.

* In Stein, we are already ready use alembic to manage DB schema changes 
in placement.

Question
========

How should we copy the resource provider id to root provider id if the 
root provider id is None?

Options
=======

1. Do it in the alembic script in the same way as the schema expansion
     * This is done in the https://review.openstack.org/#/c/619126/ and 
brought several concerns.
         * We don't want the data manipulation migration to be 
inter-mixed with schema changes.
         * For cases skipping one release in an FFU fashion, there would 
be a lot of rows to be changed.

2. Have two stable version branches for alembic to separate schema 
changes and data manipulation
     * Neutron has been managing two branches in alembic already.
         * 
https://specs.openstack.org/openstack/neutron-specs/specs/liberty/online-schema-migrations.html
     * Developers would specify on which branch to have a new version 
via some a new CLI something like; `placement-db-manage revision -m 
"description of revision" (--schema-change|--data-manipulate)`
     * We should be careful for the dependency management across the two 
branches.
	* https://alembic.sqlalchemy.org/en/latest/branches.html

3. Bring the online-migration batch CLI command
     * Follow the traditional way to manage DB data in Nova
         * `placement-manage db online-data-migration [--max-count]`
         * I'm looking into this in 
https://review.openstack.org/#/c/624942/.

4. Any other ideas?

---


Since it looks like it's going to have an impact both on operators who 
upgrade and developers who add new features in placement, it is nice to 
seek more ideas and to have a consensus before we go further.

Thanks,

-- 
Tetsuro Nakamura <nakamura.tetsuro at lab.ntt.co.jp>
NTT Network Service Systems Laboratories
TEL:0422 59 6914(National)/+81 422 59 6914(International)
3-9-11, Midori-Cho Musashino-Shi, Tokyo 180-8585 Japan