<html><head></head><body><div>On Fri, 2021-08-13 at 23:55 +0530, Abhishek Kekane wrote:</div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif">Now glance is hit with different SQLA related error, We are tracking it here [1].</div></div></blockquote><div><br></div><div>This had the same underlying cause as the previous change. I've pushed a patch to address this [1]</div><div><br></div><div>Stephen</div><div><br></div><div>[1] <a href="https://review.opendev.org/c/openstack/glance/+/804699">https://review.opendev.org/c/openstack/glance/+/804699</a></div><div><br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif">[1] <a href="https://bugs.launchpad.net/glance/+bug/1939716">https://bugs.launchpad.net/glance/+bug/1939716</a></div><div class="gmail_default" style="font-family:verdana,sans-serif"><br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Than<span class="gmail_default" style="font-family:verdana,sans-serif"></span>ks & Best Regards,<br><br></div>Abhishek Kekane<br></div></div></div><br></div><div><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 13, 2021 at 5:27 AM Mike Bayer <<a href="mailto:mike_mp@zzzcomputing.com">mike_mp@zzzcomputing.com</a>> wrote:<br></div><blockquote type="cite" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div><u></u></div><div><div>didn't see the main docs linked so the story starts at SQLA's migration docs.<br></div><div><br></div><div>starting w/ 1.4 which has all of 2.0's internals ready to go:<br></div><div><br></div><div><a href="https://docs.sqlalchemy.org/en/14/changelog/migration_14.html" target="_blank">https://docs.sqlalchemy.org/en/14/changelog/migration_14.html</a><br></div><div><br></div><div>then 2.0 which includes 1.x->2.0 migration notes:<br></div><div><br></div><div><a href="https://docs.sqlalchemy.org/en/14/changelog/migration_20.html" target="_blank">https://docs.sqlalchemy.org/en/14/changelog/migration_20.html</a><br></div><div><br></div><div>we spent a lot of time trying to make this process smooth. it has both a few ideas from how py2->py3 worked (warnings mode) and also does some things intentionally the opposite of how py2/3 did it (you can run a codebase that is compatible with 1.4 and 2.0 at the same time).<br></div><div><br></div><div><br></div><div><br></div><div><br></div><div>On Thu, Aug 12, 2021, at 12:00 PM, Stephen Finucane wrote:<br></div><blockquote type="cite" id="gmail-m_497051906796628122qt" style="margin:0 0 0 .8ex; border-left:2px #729fcf solid;padding-left:1ex"><div>tl;dr: If you haven't already started looking at SQLAlchemy 2.0 compatibility<br></div><div>and/or are still using sqlalchemy-migrate, you probably have work to do.<br></div><div><br></div><div>As you can guess from $subject, there's a new major version of SQLAlchemy in the<br></div><div>pipeline. The good news: it cleans up a lot of cruft and makes a lot of things<br></div><div>such as session management more explicit and verbose, with the idea being that<br></div><div>this should avoid a lot of gotchas and foot-gun moments. The bad news: the<br></div><div>changes are pretty extensive, and virtually anyone that uses SQLAlchemy, which<br></div><div>is to say every (?) OpenStack service out there, will likely have to make<br></div><div>changes.<br></div><div><br></div><div>This change will likely have a couple of impacts for users:<br></div><div><br></div><div># Breaking changes in oslo.db<br></div><div><br></div><div>The oslo devs have been steadily working through the many errors that oslo.db is<br></div><div>currently emitting when SQLAlchemy 2.0 deprecation warnings are turned on. The<br></div><div>patches (which is not yet complete) to address these can be found here [1]. As a<br></div><div>result of these changes, a number of oslo.db APIs are likely to start performing<br></div><div>slightly differently, and many more may be deprecated and eventually removed. An<br></div><div>example of the kind of change you can expect to see can be found in this glance<br></div><div>patch [2]. We will work to minimize these changes but some will be unavoidable<br></div><div>once sqlalchemy 2.0 actually arrives.<br></div><div><br></div><div># Breaking changes in your own project<br></div><div><br></div><div>If oslo.db seeing breaking changes because of SQLAlchemy 2.0, you can be sure<br></div><div>that you're going to see some too. Fortunately, there are ways you can get ahead<br></div><div>of this. Turning SADeprecationWarning warnings into errors for your own module,<br></div><div>as we've done for oslo.db [3][4] seems a sensible pattern to weed out these<br></div><div>issues.<br></div><div><br></div><div>As an aside, enabling warnings as errors seems to be a generally good idea in a<br></div><div>unit test environment. It's usually a lot easier to address these things as they<br></div><div>pop up that when things are finally removed and stuff explodes. To each their<br></div><div>own though, of course.<br></div><div><br></div><div># The long prophesied death of sqlalchemy-migrate<br></div><div><br></div><div>The lack of maintenance on sqlalchemy-migrate has already been well documented<br></div><div>[5], however, SQLAlchemy 2.0 is likely to be the thing that finally puts the<br></div><div>nail in sqlalchemy-migrate's coffin. I've been working on migrating the last few<br></div><div>laggards still using sqlalchemy-migrate and have successfully removed all traces<br></div><div>of it from glance (thanks, glance-core!), while the nova and cinder patches are<br></div><div>making reasonable progress (though I do wish it was faster). The only remaining<br></div><div>"core" project to be migrated is keystone. This proved a little too complicated<br></div><div>for me to do in the limited time I have to work on these things, and I'm in the<br></div><div>middle of a role change in my day job so my time to work on upstream OpenStack<br></div><div>will be decreasing much further going forward (though not to zero, I'm hoping).<br></div><div>Fortunately, there's quite a bit of prior art available now that people can<br></div><div>follow when migrating stuff [6].<br></div><div><br></div><div>Related to the oslo.db changes: everything related to sqlalchemy-migrate in<br></div><div>oslo.db should be deprecated by the next oslo.db release, and I suspect we'll be<br></div><div>pretty aggressive in pruning these. Based on codesearch.o.o, this will have<br></div><div>impacts for at least keystone and cinder.<br></div><div><br></div><div># A chance to evaluate all things DB<br></div><div><br></div><div>One positive out of this is that the changes necessary may be broad enough to<br></div><div>take the opportunity to re-evaluate decisions made regarding your DB layer.<br></div><div>We've been doing this in nova, moving lots of things about to clear up the<br></div><div>distinction between the main and API DBs and to reduce lots of tech debt that<br></div><div>had built up in 'nova.db'. Consider using the opportunity to do the if possible.<br></div><div><br></div><div>---<br></div><div><br></div><div>Hopefully this has been useful. If you have questions about any of the above,<br></div><div>please reach out and I'll do my best to answer them.<br></div><div><br></div><div>Cheers,<br></div><div>Stephen<br></div><div><br></div><div><br></div><div>[1] <a href="https://review.opendev.org/q/topic:%2522sqlalchemy-20%2522+(status:open+OR+status:merged)+project:openstack/oslo.db" target="_blank">https://review.opendev.org/q/topic:%2522sqlalchemy-20%2522+(status:open+OR+status:merged)+project:openstack/oslo.db</a><br></div><div>[2] <a href="https://review.opendev.org/c/openstack/glance/+/804406" target="_blank">https://review.opendev.org/c/openstack/glance/+/804406</a><br></div><div>[3] <a href="https://github.com/openstack/oslo.db/commit/40bce5a2baf75dc87dd591e0f71a00f221a2ba92" target="_blank">https://github.com/openstack/oslo.db/commit/40bce5a2baf75dc87dd591e0f71a00f221a2ba92</a><br></div><div>[4] <a href="https://github.com/openstack/oslo.db/commit/4c1eb966c08d29214c1905e74965f4109f41b013" target="_blank">https://github.com/openstack/oslo.db/commit/4c1eb966c08d29214c1905e74965f4109f41b013</a><br></div><div>[5] <a href="http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020672.html" target="_blank">http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020672.html</a><br></div><div>[6] <a href="https://review.opendev.org/q/topic:%22bp%252Fremove-sqlalchemy-migrate%22+(status:open%20OR%20status:merged)" target="_blank">https://review.opendev.org/q/topic:%22bp%252Fremove-sqlalchemy-migrate%22+(status:open%20OR%20status:merged)</a><br></div><div><br></div><div><br></div><div><br></div></blockquote><div><br></div></div></blockquote></div></blockquote><div><br></div><div><span></span></div></body></html>