Open Stack

On Mon, Apr 14, 2014 at 10:54 AM, Salvatore Orlando <sorlando at nicira.com> wrote:
> Resending with [Neutron] tag.
>
> Salvatore
>
>
> On 14 April 2014 16:00, Salvatore Orlando <sorlando at nicira.com> wrote:
>>
>> This is a rather long post. However the gist of it is that Neutron
>> migrations are failing to correctly perform database upgrades when service
>> plugins are involved, and this probably means the conditional migration path
>> we designed for the Grizzly release is proving not robust enough when
>> dealing with service plugins.
>>
>> Corrective actions are therefore required; otherwise there is a risk the
>> Grenade jobs for neutron can't be made voting again.
>> This issue is already being tracked with bug 1307344 which was
>> independently filed and amended by Sean to reflect the actual issue.
>>
>> Grenade is currently failing because of this issue. Indeed the 'metering
>> plugin' is enabled in devstack for Icehouse but now Havana.
>> A quick fix for grenade has already been rejected (backport metering
>> support to stable/havana) another quick fix instead would be running grenade
>> without metering at all.
>>
>> However, whether we put or not a patch to make grenade running, there is
>> still an issue that needs to be solved.
>> Now, if you're not yet bored you might try now reading the long version of
>> this post...
>>
>> Problem statement:
>>
>> Enabling or disabling a service plugin in neutron configuration might be
>> done at any point and can lead migrations to build a database which is out
>> of sync with expectations
>>
>> Assume:
>>
>> Current migration X (Service plugin SP enabled here)
>> Service plugin SP introduced at migration F1 < X
>> No auto-generation of DB schemas
>> The migration F1 will be skipped
>>
>> When SP is enabled, the relevant DB schema won't exist and the feature
>> will not work.
>>
>> Other failure scenarios:
>>
>> Current migration X
>> SP introduced at migration F1 < X (but not yet enabled)
>> SP enabled at migration Y > F1 > X
>> SP updated at migration F2 | X < F2 < Y.
>>
>> F2 will fail because schemas are not yet ready for feature, even with auto
>> schema generation
>>
>> SP introduced at migration F0, and enabled at migration F0
>> SP disabled at migration X
>> SP updated at migration F1, which is skipped because it's disabled
>> SP enabled again at migration Y. Feature won't work as migration X < F1 <
>> Y was never executed.
>>
>> Requirements for a possible solution:
>>
>> Should solve problem not only for greenfield deployments but also for
>> existing ones
>> Must not make migration generation more complex or introduce error-prone
>> behaviours
>>
>> Possible solutions:
>>
>> 1) Specify that all migrations must run for every plugin (*) unless they
>> are really introducing schemas which are specific to a particular technology
>> (such as uuid mappings between neutron and backed)
>>
>> This approach will probably ensure all the important migrations are
>> executed.
>> However, it is not back portable, unless deployers decide to tear down and
>> rebuild their databases from scratch, which is unlikely to happen.
>>
>> To this aim "recovery" scripts might be provided to sync up the schema
>> state of specific service plugins with the current alembic migration
>> registered in the neutron database; or appropriate migrations can be added
>> in the path to fix database schemas.
>>
>> (*) Neutron does not address, and probably never will address, switching
>> from plugin "a" to "b" for a given service (e.g.: core features).
>>
>> 2) Have plugin-specific migration paths.
>>
>> This is not supported by alembic. In this case probably the best solution
>> would be to have completely different databases for the main plugin and
>> service plugins, which is however not possible because of foreign key
>> relationships among tables in different service plugins.
>>
>> Also, there are likely to be migrations which alter core and service
>> schemas at the same time. Adapting these migrations might be a problem.Extra
>> work will also be needed to make this solution work with existing
>> deployments.
>>
>> 3) Make migrations smarter
>>
>> When migration X is executed, it checks whether any previous migration for
>> the plugins it alters has been executed. If not it will execute the missing
>> migrations as well.
>> This *might* work; However, alembic currently offers no support for it, as
>> it's not possible to perform any form of "introspection" in the migration
>> themselves.
>>
>> The migration engine will also be augmented as the last executed migration
>> for each plugin should be stored in the database.
>> This might also prove to be error prone and lead to unexpected behaviours.
>> For example:
>>
>> Until migration X deployment uses plugin A which provides services S1 and
>> S2
>> At migration Y user still uses plugin for service S1, but switches to
>> plugin B service S2
>> Migration Z>Y applies to plugin B. System detects no migration for plugin
>> B have been executed so far.
>> However, the migrations for the service S2, offered by B were already
>> executed when running plugin A
>> System tries to execute migrations for plugin B which will fail as they
>> were actually already executed for plugin A
>>
>> The system could be made smarter by storing also a list of "known"
>> migrations, including whether they were executed or skipped.
>>
>> Summarising, in my opinion the approach #2 is not even worth being
>> considered. Between approach #1 and #3, I'd prefer the simplest one and vote
>> for approach #1.
>>

Thanks for sending out this summary email Salvatore. I agree with your
summary and I think that option #1 is the way forward to fix this. The
recovery scripts are a necessity here. I'd like to hear from some
deployers who may be lurking on their thoughts on this approach as
well.

Thanks,
Kyle

>>
>> Salvatore
>>
>>
>>
>>
>>
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>