Open Stack

Hi Salvatore,

The described problem could be even worse if vendor drivers are considered.
Doesn't #1 require that all DB tables are named differently? Otherwise it
seems that user can't be sure in DB schema even if all tables are present.

I think the big part of the problem is that we need to support both online
and offline migrations. Without the latter things could be a little bit
simpler.

Also it seems to me that problem statement should be changed to the
following:
One need to migrate from (Config1, MigrationID1) to (Config2,
MigrationID2), and currently our code only accounts for MigrationIDs.
We may consider amending DB with configuration metadata, at least that will
allow to run migration code with full knowledge of what happened before (if
online mode is considered).
In offline mode that will require providing old and current configurations.

That was just thinking aloud, no concrete proposals yet.

Thanks,
Eugene.



On Mon, Apr 14, 2014 at 7:00 PM, Salvatore Orlando <sorlando at nicira.com>wrote:

> This is a rather long post. However the gist of it is that Neutron
> migrations are failing to correctly perform database upgrades when service
> plugins are involved, and this probably means the conditional migration
> path we designed for the Grizzly release is proving not robust enough when
> dealing with service plugins.
>
> Corrective actions are therefore required; otherwise there is a risk the
> Grenade jobs for neutron can't be made voting again.
> This issue is already being tracked with bug 1307344 which was
> independently filed and amended by Sean to reflect the actual issue.
>
> Grenade is currently failing because of this issue. Indeed the 'metering
> plugin' is enabled in devstack for Icehouse but now Havana.
> A quick fix for grenade has already been rejected (backport metering
> support to stable/havana) another quick fix instead would be running
> grenade without metering at all.
>
> However, whether we put or not a patch to make grenade running, there is
> still an issue that needs to be solved.
> Now, if you're not yet bored you might try now reading the long version of
> this post...
>
> Problem statement:
>
> Enabling or disabling a service plugin in neutron configuration might be
> done at any point and can lead migrations to build a database which is out
> of sync with expectations
>
> Assume:
>
> Current migration X (Service plugin SP enabled here)
> Service plugin SP introduced at migration F1 < X
> No auto-generation of DB schemas
> The migration F1 will be skipped
>
> When SP is enabled, the relevant DB schema won't exist and the feature
> will not work.
>
> Other failure scenarios:
>
> Current migration X
> SP introduced at migration F1 < X (but not yet enabled)
> SP enabled at migration Y > F1 > X
> SP updated at migration F2 | X < F2 < Y.
>
> F2 will fail because schemas are not yet ready for feature, even with auto
> schema generation
>
> SP introduced at migration F0, and enabled at migration F0
> SP disabled at migration X
> SP updated at migration F1, which is skipped because it's disabled
> SP enabled again at migration Y. Feature won't work as migration X < F1 <
> Y was never executed.
>
> Requirements for a possible solution:
>
> Should solve problem not only for greenfield deployments but also for
> existing ones
> Must not make migration generation more complex or introduce error-prone
> behaviours
>
> Possible solutions:
>
> 1) Specify that all migrations must run for every plugin (*) unless they
> are really introducing schemas which are specific to a particular
> technology (such as uuid mappings between neutron and backed)
>
> This approach will probably ensure all the important migrations are
> executed.
> However, it is not back portable, unless deployers decide to tear down and
> rebuild their databases from scratch, which is unlikely to happen.
>
> To this aim "recovery" scripts might be provided to sync up the schema
> state of specific service plugins with the current alembic migration
> registered in the neutron database; or appropriate migrations can be added
> in the path to fix database schemas.
>
> (*) Neutron does not address, and probably never will address, switching
> from plugin "a" to "b" for a given service (e.g.: core features).
>
> 2) Have plugin-specific migration paths.
>
> This is not supported by alembic. In this case probably the best solution
> would be to have completely different databases for the main plugin and
> service plugins, which is however not possible because of foreign key
> relationships among tables in different service plugins.
>
> Also, there are likely to be migrations which alter core and service
> schemas at the same time. Adapting these migrations might be a
> problem.Extra work will also be needed to make this solution work with
> existing deployments.
>
> 3) Make migrations smarter
>
> When migration X is executed, it checks whether any previous migration for
> the plugins it alters has been executed. If not it will execute the missing
> migrations as well.
> This *might* work; However, alembic currently offers no support for it, as
> it's not possible to perform any form of "introspection" in the migration
> themselves.
>
> The migration engine will also be augmented as the last executed migration
> for each plugin should be stored in the database.
> This might also prove to be error prone and lead to unexpected behaviours.
> For example:
>
> Until migration X deployment uses plugin A which provides services S1 and
> S2
> At migration Y user still uses plugin for service S1, but switches to
> plugin B service S2
> Migration Z>Y applies to plugin B. System detects no migration for plugin
> B have been executed so far.
> However, the migrations for the service S2, offered by B were already
> executed when running plugin A
> System tries to execute migrations for plugin B which will fail as they
> were actually already executed for plugin A
>
> The system could be made smarter by storing also a list of "known"
> migrations, including whether they were executed or skipped.
>
> Summarising, in my opinion the approach #2 is not even worth being
> considered. Between approach #1 and #3, I'd prefer the simplest one and
> vote for approach #1.
>
>
> Salvatore
>
>
>
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140414/de8bbe8a/attachment.html>