[openstack-dev] [keystone][nova][neutron][all] Rolling upgrades: database triggers and oslo.versionedobjects

David Stanek dstanek at dstanek.com
Thu Sep 1 13:57:40 UTC 2016


On Wed, Aug 31 at 17:18 -0500, Monty Taylor wrote:
> 
> Nova and Neutron have an approach for this. It may or may not be ideal -
> but it exists right now. While it can be satisfying to discount the
> existing approach and write a new one, I do not believe that is in the
> best interests of OpenStack as a whole. To diverge in _keystone_ - which
> is one of the few projects that must exist in every OpenStack install -
> when there exists an approach in the two other most commonly deployed
> projects - is such a terrible example of the problems inherent in
> Conway's Law that it makes me want to push up a proposal to dissolve all
> of the individual project teams and merge all of the repos into a single
> repo.

That's a bit overly dramatic. I think having some innovation is a good
thing. Specifically in this case where our needs appear to be a little
simpler than those of nova.

> 
> Make the oslo libraries Nova and Neutron are using better. Work with the
> Nova and Neutron teams on a consolidated approach. We need to be driving
> more towards an OpenStack that behaves as if it wasn't written by
> warring factions of developers who barely communicate.

I believe we tried to keep with the same extract/migrate/contract
patterns. Sure our implementation differs, but I don't see operators
caring about that as long as it works.

> 
> Even if the idea was one I thought was good technically, the above would
> still trump that. Work with Nova and Neutron. Be more similar.
> 
> PLEASE
> 
> BUT - I also don't think it's a good technical solution. That isn't
> because triggers don't work in MySQL (they do) - but because we've spent
> the last six years explicitly NOT writing raw SQL. We've chosen an
> abstraction layer (SQLAlchemy) which does its job well.
> 
> IF this were going to be accompanied by a corresponding shift in
> approach to not support any backends by MySQL and to start writing our
> database interactions directly in SQL in ALL of our projects - I could
> MAYBE be convinced. Even then I think doing it in triggers is the wrong
> place to put logic.
> 
> "Database triggers are obviously a new challenge for developers to
> write, honestly challenging to debug (being side effects), and are made
> even more difficult by having to hand write triggers for MySQL,
> PostgreSQL, and SQLite independently (SQLAlchemy offers no assistance in
> this case)"
> 
> If you look at:
> 
> https://review.openstack.org/#/c/355618/40/keystone/common/sql/expand_repo/versions/002_add_key_hash_and_encrypted_blob_to_credential.py
> 
> You will see the three different SQL dialects this. Not only that, but
> some of the more esoteric corners of those backends. We can barely get
> _indexes_ right in our database layers ... now we think we're going to
> get triggers right? Consistently? And handle things like Galera?
> 
> The other option is app level, which is what nova and neutron are doing.
> It's a good option, because it puts the logic in python, which is a
> thing we have 2500 developers fairly well versed in. It's also scalable,
> as the things executing whatever the logic is are themselves a scale-out
> set of servers. Finally, it's a known and accepted pattern in large
> scale MySQL shops ... Roll out a new version of the app code which
> understands both the old and the new schema version, then roll out a
> no-downtime additive schema change to the database, then have the app
> layer process and handle on the fly transformation if needed.

I've done both types of migrations in the past, but with one imporant
exception. We could roll out our application on Tuesday and then the
cleanup on Thursday. We didn't carry baggage for 6 months to a year. My
fear with keystone is that we'd slow development even more by adding
more cruft and cruft on top of cuft.

> 
> SO ...
> 
> Just do what Nova and Neutron are doing - and if it's not good enough,
> fix it. Having some projects use triggers and other projects not use
> triggers is one of the more epically crazypants things I've heard around
> here ... and I lived through the twisted/eventlet argument.

-- 
David Stanek
web: http://dstanek.com
blog: http://traceback.org



More information about the OpenStack-dev mailing list