Open Stack

Resending with [Neutron] tag.

Salvatore


On 14 April 2014 16:00, Salvatore Orlando <sorlando at nicira.com> wrote:

> This is a rather long post. However the gist of it is that Neutron
> migrations are failing to correctly perform database upgrades when service
> plugins are involved, and this probably means the conditional migration
> path we designed for the Grizzly release is proving not robust enough when
> dealing with service plugins.
>
> Corrective actions are therefore required; otherwise there is a risk the
> Grenade jobs for neutron can't be made voting again.
> This issue is already being tracked with bug 1307344 which was
> independently filed and amended by Sean to reflect the actual issue.
>
> Grenade is currently failing because of this issue. Indeed the 'metering
> plugin' is enabled in devstack for Icehouse but now Havana.
> A quick fix for grenade has already been rejected (backport metering
> support to stable/havana) another quick fix instead would be running
> grenade without metering at all.
>
> However, whether we put or not a patch to make grenade running, there is
> still an issue that needs to be solved.
> Now, if you're not yet bored you might try now reading the long version of
> this post...
>
> Problem statement:
>
> Enabling or disabling a service plugin in neutron configuration might be
> done at any point and can lead migrations to build a database which is out
> of sync with expectations
>
> Assume:
>
> Current migration X (Service plugin SP enabled here)
> Service plugin SP introduced at migration F1 < X
> No auto-generation of DB schemas
> The migration F1 will be skipped
>
> When SP is enabled, the relevant DB schema won't exist and the feature
> will not work.
>
> Other failure scenarios:
>
> Current migration X
> SP introduced at migration F1 < X (but not yet enabled)
> SP enabled at migration Y > F1 > X
> SP updated at migration F2 | X < F2 < Y.
>
> F2 will fail because schemas are not yet ready for feature, even with auto
> schema generation
>
> SP introduced at migration F0, and enabled at migration F0
> SP disabled at migration X
> SP updated at migration F1, which is skipped because it's disabled
> SP enabled again at migration Y. Feature won't work as migration X < F1 <
> Y was never executed.
>
> Requirements for a possible solution:
>
> Should solve problem not only for greenfield deployments but also for
> existing ones
> Must not make migration generation more complex or introduce error-prone
> behaviours
>
> Possible solutions:
>
> 1) Specify that all migrations must run for every plugin (*) unless they
> are really introducing schemas which are specific to a particular
> technology (such as uuid mappings between neutron and backed)
>
> This approach will probably ensure all the important migrations are
> executed.
> However, it is not back portable, unless deployers decide to tear down and
> rebuild their databases from scratch, which is unlikely to happen.
>
> To this aim "recovery" scripts might be provided to sync up the schema
> state of specific service plugins with the current alembic migration
> registered in the neutron database; or appropriate migrations can be added
> in the path to fix database schemas.
>
> (*) Neutron does not address, and probably never will address, switching
> from plugin "a" to "b" for a given service (e.g.: core features).
>
> 2) Have plugin-specific migration paths.
>
> This is not supported by alembic. In this case probably the best solution
> would be to have completely different databases for the main plugin and
> service plugins, which is however not possible because of foreign key
> relationships among tables in different service plugins.
>
> Also, there are likely to be migrations which alter core and service
> schemas at the same time. Adapting these migrations might be a
> problem.Extra work will also be needed to make this solution work with
> existing deployments.
>
> 3) Make migrations smarter
>
> When migration X is executed, it checks whether any previous migration for
> the plugins it alters has been executed. If not it will execute the missing
> migrations as well.
> This *might* work; However, alembic currently offers no support for it, as
> it's not possible to perform any form of "introspection" in the migration
> themselves.
>
> The migration engine will also be augmented as the last executed migration
> for each plugin should be stored in the database.
> This might also prove to be error prone and lead to unexpected behaviours.
> For example:
>
> Until migration X deployment uses plugin A which provides services S1 and
> S2
> At migration Y user still uses plugin for service S1, but switches to
> plugin B service S2
> Migration Z>Y applies to plugin B. System detects no migration for plugin
> B have been executed so far.
> However, the migrations for the service S2, offered by B were already
> executed when running plugin A
> System tries to execute migrations for plugin B which will fail as they
> were actually already executed for plugin A
>
> The system could be made smarter by storing also a list of "known"
> migrations, including whether they were executed or skipped.
>
> Summarising, in my opinion the approach #2 is not even worth being
> considered. Between approach #1 and #3, I'd prefer the simplest one and
> vote for approach #1.
>
>
> Salvatore
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140414/6739f514/attachment.html>