[openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

Flavio Percoco flavio at redhat.com
Tue Aug 30 08:42:55 UTC 2016

On 25/08/16 13:13 -0400, Steve Martinelli wrote:
>The keystone team is pursuing a trigger-based approach to support rolling,
>zero-downtime upgrades. The proposed operator experience is documented here:
>  http://docs.openstack.org/developer/keystone/upgrading.html
>This differs from Nova and Neutron's approaches to solve for rolling
>upgrades (which use oslo.versionedobjects), however Keystone is one of the
>few services that doesn't need to manage communication between multiple
>releases of multiple service components talking over the message bus (which
>is the original use case for oslo.versionedobjects, and for which it is
>aptly suited). Keystone simply scales horizontally and every node talks
>directly to the database.
>Database triggers are obviously a new challenge for developers to write,
>honestly challenging to debug (being side effects), and are made even more
>difficult by having to hand write triggers for MySQL, PostgreSQL, and
>SQLite independently (SQLAlchemy offers no assistance in this case), as
>seen in this patch:
>  https://review.openstack.org/#/c/355618/
>However, implementing an application-layer solution with
>oslo.versionedobjects is not an easy task either; refer to Neutron's
>Our primary concern at this point are how to effectively test the triggers
>we write against our supported database systems, and their various
>deployment variations. We might be able to easily drop SQLite support (as
>it's only supported for our own test suite), but should we expect variation
>in support and/or actual behavior of triggers across the MySQLs, MariaDBs,
>Perconas, etc, of the world that would make it necessary to test each of
>them independently? If you have operational experience working with
>triggers at scale: are there landmines that we need to be aware of? What is
>it going to take for us to say we support *zero* dowtime upgrades with

Hey Steve, Dolph,

Thanks for sending this out. There's been some discussions in the Glance
community about how we can implement rolling upgrades and it seems like Glance's
case is very similar to keystone's.

I'll make sure folks in the glance community are aware of this thread and reach


Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 847 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160830/67c526ff/attachment.pgp>

More information about the OpenStack-dev mailing list