Open Stack

This is a rather long post. However the gist of it is that Neutron
migrations are failing to correctly perform database upgrades when service
plugins are involved, and this probably means the conditional migration
path we designed for the Grizzly release is proving not robust enough when
dealing with service plugins.

Corrective actions are therefore required; otherwise there is a risk the
Grenade jobs for neutron can't be made voting again.
This issue is already being tracked with bug 1307344 which was
independently filed and amended by Sean to reflect the actual issue.

Grenade is currently failing because of this issue. Indeed the 'metering
plugin' is enabled in devstack for Icehouse but now Havana.
A quick fix for grenade has already been rejected (backport metering
support to stable/havana) another quick fix instead would be running
grenade without metering at all.

However, whether we put or not a patch to make grenade running, there is
still an issue that needs to be solved.
Now, if you're not yet bored you might try now reading the long version of
this post...

Problem statement:

Enabling or disabling a service plugin in neutron configuration might be
done at any point and can lead migrations to build a database which is out
of sync with expectations

Assume:

Current migration X (Service plugin SP enabled here)
Service plugin SP introduced at migration F1 < X
No auto-generation of DB schemas
The migration F1 will be skipped

When SP is enabled, the relevant DB schema won't exist and the feature will
not work.

Other failure scenarios:

Current migration X
SP introduced at migration F1 < X (but not yet enabled)
SP enabled at migration Y > F1 > X
SP updated at migration F2 | X < F2 < Y.

F2 will fail because schemas are not yet ready for feature, even with auto
schema generation

SP introduced at migration F0, and enabled at migration F0
SP disabled at migration X
SP updated at migration F1, which is skipped because it's disabled
SP enabled again at migration Y. Feature won't work as migration X < F1 < Y
was never executed.

Requirements for a possible solution:

Should solve problem not only for greenfield deployments but also for
existing ones
Must not make migration generation more complex or introduce error-prone
behaviours

Possible solutions:

1) Specify that all migrations must run for every plugin (*) unless they
are really introducing schemas which are specific to a particular
technology (such as uuid mappings between neutron and backed)

This approach will probably ensure all the important migrations are
executed.
However, it is not back portable, unless deployers decide to tear down and
rebuild their databases from scratch, which is unlikely to happen.

To this aim "recovery" scripts might be provided to sync up the schema
state of specific service plugins with the current alembic migration
registered in the neutron database; or appropriate migrations can be added
in the path to fix database schemas.

(*) Neutron does not address, and probably never will address, switching
from plugin "a" to "b" for a given service (e.g.: core features).

2) Have plugin-specific migration paths.

This is not supported by alembic. In this case probably the best solution
would be to have completely different databases for the main plugin and
service plugins, which is however not possible because of foreign key
relationships among tables in different service plugins.

Also, there are likely to be migrations which alter core and service
schemas at the same time. Adapting these migrations might be a
problem.Extra work will also be needed to make this solution work with
existing deployments.

3) Make migrations smarter

When migration X is executed, it checks whether any previous migration for
the plugins it alters has been executed. If not it will execute the missing
migrations as well.
This *might* work; However, alembic currently offers no support for it, as
it's not possible to perform any form of "introspection" in the migration
themselves.

The migration engine will also be augmented as the last executed migration
for each plugin should be stored in the database.
This might also prove to be error prone and lead to unexpected behaviours.
For example:

Until migration X deployment uses plugin A which provides services S1 and S2
At migration Y user still uses plugin for service S1, but switches to
plugin B service S2
Migration Z>Y applies to plugin B. System detects no migration for plugin B
have been executed so far.
However, the migrations for the service S2, offered by B were already
executed when running plugin A
System tries to execute migrations for plugin B which will fail as they
were actually already executed for plugin A

The system could be made smarter by storing also a list of "known"
migrations, including whether they were executed or skipped.

Summarising, in my opinion the approach #2 is not even worth being
considered. Between approach #1 and #3, I'd prefer the simplest one and
vote for approach #1.


Salvatore
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140414/613faf44/attachment.html>