Open Stack

tl;dr: If you haven't already started looking at SQLAlchemy 2.0 compatibility
and/or are still using sqlalchemy-migrate, you probably have work to do.

As you can guess from $subject, there's a new major version of SQLAlchemy in the
pipeline. The good news: it cleans up a lot of cruft and makes a lot of things
such as session management more explicit and verbose, with the idea being that
this should avoid a lot of gotchas and foot-gun moments. The bad news: the
changes are pretty extensive, and virtually anyone that uses SQLAlchemy, which
is to say every (?) OpenStack service out there, will likely have to make
changes.

This change will likely have a couple of impacts for users:

# Breaking changes in oslo.db

The oslo devs have been steadily working through the many errors that oslo.db is
currently emitting when SQLAlchemy 2.0 deprecation warnings are turned on. The
patches (which is not yet complete) to address these can be found here [1]. As a
result of these changes, a number of oslo.db APIs are likely to start performing
slightly differently, and many more may be deprecated and eventually removed. An
example of the kind of change you can expect to see can be found in this glance
patch [2]. We will work to minimize these changes but some will be unavoidable
once sqlalchemy 2.0 actually arrives.

# Breaking changes in your own project

If oslo.db seeing breaking changes because of SQLAlchemy 2.0, you can be sure
that you're going to see some too. Fortunately, there are ways you can get ahead
of this. Turning SADeprecationWarning warnings into errors for your own module,
as we've done for oslo.db [3][4] seems a sensible pattern to weed out these
issues.

As an aside, enabling warnings as errors seems to be a generally good idea in a
unit test environment. It's usually a lot easier to address these things as they
pop up that when things are finally removed and stuff explodes. To each their
own though, of course.

# The long prophesied death of sqlalchemy-migrate

The lack of maintenance on sqlalchemy-migrate has already been well documented
[5], however, SQLAlchemy 2.0 is likely to be the thing that finally puts the
nail in sqlalchemy-migrate's coffin. I've been working on migrating the last few
laggards still using sqlalchemy-migrate and have successfully removed all traces
of it from glance (thanks, glance-core!), while the nova and cinder patches are
making reasonable progress (though I do wish it was faster). The only remaining
"core" project to be migrated is keystone. This proved a little too complicated
for me to do in the limited time I have to work on these things, and I'm in the
middle of a role change in my day job so my time to work on upstream OpenStack
will be decreasing much further going forward (though not to zero, I'm hoping).
Fortunately, there's quite a bit of prior art available now that people can
follow when migrating stuff [6].

Related to the oslo.db changes: everything related to sqlalchemy-migrate in
oslo.db should be deprecated by the next oslo.db release, and I suspect we'll be
pretty aggressive in pruning these. Based on codesearch.o.o, this will have
impacts for at least keystone and cinder.

# A chance to evaluate all things DB

One positive out of this is that the changes necessary may be broad enough to
take the opportunity to re-evaluate decisions made regarding your DB layer.
We've been doing this in nova, moving lots of things about to clear up the
distinction between the main and API DBs and to reduce lots of tech debt that
had built up in 'nova.db'. Consider using the opportunity to do the if possible.

---

Hopefully this has been useful. If you have questions about any of the above,
please reach out and I'll do my best to answer them.

Cheers,
Stephen


[1] https://review.opendev.org/q/topic:%2522sqlalchemy-20%2522+(status:open+OR+status:merged)+project:openstack/oslo.db
[2] https://review.opendev.org/c/openstack/glance/+/804406
[3] https://github.com/openstack/oslo.db/commit/40bce5a2baf75dc87dd591e0f71a00f221a2ba92
[4] https://github.com/openstack/oslo.db/commit/4c1eb966c08d29214c1905e74965f4109f41b013
[5] http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020672.html
[6] https://review.opendev.org/q/topic:%22bp%252Fremove-sqlalchemy-migrate%22+(status:open%20OR%20status:merged)