Need core reviewers for sqlalchemy-migrate
Hi, Left, there's only myself and Monty as core reviewers. I don't have the skills, or time for this. The only reason I'm a core reviewer, is that historically, I was the one pointing out it got unmaintained, back a long time ago (many years from now). So it'd be nice of some people could step up and become core reviewer of SQLA-migrate please. Cheers, Thomas Goirand (zigo)
On Mon, Feb 22, 2021 at 09:21:18PM +0100, Thomas Goirand wrote:
Hi,
Left, there's only myself and Monty as core reviewers. I don't have the skills, or time for this. The only reason I'm a core reviewer, is that historically, I was the one pointing out it got unmaintained, back a long time ago (many years from now).
So it'd be nice of some people could step up and become core reviewer of SQLA-migrate please.
I may have things mixed up now, but I thought I was told a few years ago now that sqlalchemy-migrate was deprecated and that projects should be migrating to alembic now. Maybe someone that knows more about this can comment? Sean
On 2/23/2021 11:11 AM, Sean McGinnis wrote:
On Mon, Feb 22, 2021 at 09:21:18PM +0100, Thomas Goirand wrote:
Hi,
Left, there's only myself and Monty as core reviewers. I don't have the skills, or time for this. The only reason I'm a core reviewer, is that historically, I was the one pointing out it got unmaintained, back a long time ago (many years from now).
So it'd be nice of some people could step up and become core reviewer of SQLA-migrate please.
I may have things mixed up now, but I thought I was told a few years ago now that sqlalchemy-migrate was deprecated and that projects should be migrating to alembic now.
This was my understanding as well. Has that position changed? Thanks! Jay
Maybe someone that knows more about this can comment?
Sean
On 2/23/21 11:42 AM, Jay Bryant wrote:
On 2/23/2021 11:11 AM, Sean McGinnis wrote:
On Mon, Feb 22, 2021 at 09:21:18PM +0100, Thomas Goirand wrote:
Hi,
Left, there's only myself and Monty as core reviewers. I don't have the skills, or time for this. The only reason I'm a core reviewer, is that historically, I was the one pointing out it got unmaintained, back a long time ago (many years from now).
So it'd be nice of some people could step up and become core reviewer of SQLA-migrate please.
I may have things mixed up now, but I thought I was told a few years ago now that sqlalchemy-migrate was deprecated and that projects should be migrating to alembic now.
This was my understanding as well. Has that position changed?
Quite the contrary, oslo.db just deprecated migrate support: https://opendev.org/openstack/oslo.db/commit/fc855875236fd3bf760237fc64092f4... This came up in the keystone meeting a few weeks ago because the deprecation broke their test jobs. Unfortunately, from looking at other projects' migration patches it doesn't appear to be trivial to move to alembic. Based on the commit message, I think the deprecation was more intended to prod users of migrate to move off it as opposed to start a ticking clock for its removal, but we should have a discussion about how exactly projects should proceed. Maybe a pop-up team that could write docs and provide a point of contact for people with questions?
On 2021-02-23 12:39:47 -0600 (-0600), Ben Nemec wrote:
On 2/23/21 11:42 AM, Jay Bryant wrote:
On 2/23/2021 11:11 AM, Sean McGinnis wrote: [...]
I may have things mixed up now, but I thought I was told a few years ago now that sqlalchemy-migrate was deprecated and that projects should be migrating to alembic now.
This was my understanding as well. Has that position changed?
Quite the contrary, oslo.db just deprecated migrate support: https://opendev.org/openstack/oslo.db/commit/fc855875236fd3bf760237fc64092f4...
This came up in the keystone meeting a few weeks ago because the deprecation broke their test jobs. Unfortunately, from looking at other projects' migration patches it doesn't appear to be trivial [...]
For a bit of history, the SQLAM maintainers decided to abandon the project, so it was forked into StackForge (remember StackForge?) by https://review.openstack.org/36723 in mid-2013. The commit message at the time suggested, "The overall project plan is to transition to alembic." I guess 8 years isn't enough warning to get that to happen. -- Jeremy Stanley
On 2/23/21 8:25 PM, Jeremy Stanley wrote:
On 2021-02-23 12:39:47 -0600 (-0600), Ben Nemec wrote:
On 2/23/21 11:42 AM, Jay Bryant wrote:
On 2/23/2021 11:11 AM, Sean McGinnis wrote: [...]
I may have things mixed up now, but I thought I was told a few years ago now that sqlalchemy-migrate was deprecated and that projects should be migrating to alembic now.
This was my understanding as well. Has that position changed?
Quite the contrary, oslo.db just deprecated migrate support: https://opendev.org/openstack/oslo.db/commit/fc855875236fd3bf760237fc64092f4...
This came up in the keystone meeting a few weeks ago because the deprecation broke their test jobs. Unfortunately, from looking at other projects' migration patches it doesn't appear to be trivial [...]
For a bit of history, the SQLAM maintainers decided to abandon the project, so it was forked into StackForge (remember StackForge?) by https://review.openstack.org/36723 in mid-2013. The commit message at the time suggested, "The overall project plan is to transition to alembic." I guess 8 years isn't enough warning to get that to happen.
Right! Though there was the plan to move to Alembic, as mentioned in this thread, the transition may not be easy. That original author happened to be a Debian developer, and I was the one telling the list about the situation. And that's how I became a core reviewer of the project, even if I don't even know how to write a program that would SQLAM. We still have Nova, Cinder, Heat, and probably others that still use SQLAM, so it's probably not the best idea to abandon it just yet... :) Therefore, I'm asking again: someone must step up to maintain SQLAM. Cheers, Thomas Goirand (zigo)
On Tue, 2021-02-23 at 21:22 +0100, Thomas Goirand wrote:
On 2/23/21 8:25 PM, Jeremy Stanley wrote:
On 2021-02-23 12:39:47 -0600 (-0600), Ben Nemec wrote:
On 2/23/21 11:42 AM, Jay Bryant wrote:
On 2/23/2021 11:11 AM, Sean McGinnis wrote: [...]
I may have things mixed up now, but I thought I was told a few years ago now that sqlalchemy-migrate was deprecated and that projects should be migrating to alembic now.
This was my understanding as well. Has that position changed?
Quite the contrary, oslo.db just deprecated migrate support: https://opendev.org/openstack/oslo.db/commit/fc855875236fd3bf760237fc64092f4...
This came up in the keystone meeting a few weeks ago because the deprecation broke their test jobs. Unfortunately, from looking at other projects' migration patches it doesn't appear to be trivial [...]
For a bit of history, the SQLAM maintainers decided to abandon the project, so it was forked into StackForge (remember StackForge?) by https://review.openstack.org/36723 in mid-2013. The commit message at the time suggested, "The overall project plan is to transition to alembic." I guess 8 years isn't enough warning to get that to happen.
Right!
Though there was the plan to move to Alembic, as mentioned in this thread, the transition may not be easy.
That original author happened to be a Debian developer, and I was the one telling the list about the situation. And that's how I became a core reviewer of the project, even if I don't even know how to write a program that would SQLAM.
We still have Nova, Cinder, Heat, and probably others that still use SQLAM, so it's probably not the best idea to abandon it just yet... :)
Therefore, I'm asking again: someone must step up to maintain SQLAM. if im not mistaken it used to be maintained by Matt Riedemann until he moved on form openstack last year. The nova team had felt that the maintance of sqlam was not not more then the cost of doing the transitaion so it was a low priorty task to move and matt and other just fixed SQLAM whenever it needed to be fixed.
nova is currently in the processs of droping it and adopting alembic. https://review.opendev.org/q/topic:%22bp%252Fcompact-db-migrations-wallaby%2...) i think the plan is still to complet that work before the end of the cycle. the primary db has already beeen compacted and once all the migrations are flattened we planned to use alembic for future migrations. im not sure alembic will be adopted this cycle for nova but it should be next cycle. its ok for oslo.db to deprecate sqlam support provdied they do not remove it untll all the projects have moved.
Cheers,
Thomas Goirand (zigo)
On Tue, 2021-02-23 at 12:39 -0600, Ben Nemec wrote:
On 2/23/21 11:42 AM, Jay Bryant wrote:
On 2/23/2021 11:11 AM, Sean McGinnis wrote:
On Mon, Feb 22, 2021 at 09:21:18PM +0100, Thomas Goirand wrote:
Hi,
Left, there's only myself and Monty as core reviewers. I don't have the skills, or time for this. The only reason I'm a core reviewer, is that historically, I was the one pointing out it got unmaintained, back a long time ago (many years from now).
So it'd be nice of some people could step up and become core reviewer of SQLA-migrate please.
I may have things mixed up now, but I thought I was told a few years ago now that sqlalchemy-migrate was deprecated and that projects should be migrating to alembic now.
This was my understanding as well. Has that position changed?
Quite the contrary, oslo.db just deprecated migrate support: https://opendev.org/openstack/oslo.db/commit/fc855875236fd3bf760237fc64092f4...
This came up in the keystone meeting a few weeks ago because the deprecation broke their test jobs. Unfortunately, from looking at other projects' migration patches it doesn't appear to be trivial to move to alembic.
If other projects are anything like nova, then I suspect DB migrations happen relatively infrequently now. Everyone should be able do what nova is already doing/planning to: compact everything up to a relatively recent release (we chose Train) and then a new initial migration patch in alembic lingo with the following pseudo-logic: is database already initialized/versioned with sqlalchemy-migrate? \-> yes | is latest version applied? \-> yes dummy apply alembic initial migration unless already applied \-> no tell user to upgrade \-> no | apply alembic initial migration unless already applied
Based on the commit message, I think the deprecation was more intended to prod users of migrate to move off it as opposed to start a ticking clock for its removal, but we should have a discussion about how exactly projects should proceed. Maybe a pop-up team that could write docs and provide a point of contact for people with questions?
There was that and also to give an indication that no one is maintaining the code and it therefore shouldn't be relied on (bit rot is a thing). This is pretty much the Mox-mock thing all over again. sqlalchemy-migrate needs to be retired and just needs bodies to do so. Alembic is by all accounts a much better solution and it is actively maintained by the sqlalchemy community. I'm working on removing sqlalchemy-migrate from nova, but it'll be Xena before we can really dig into that. I proposed a BP and patches to remove it from Glance early this cycle but alas that got very limited attention (thanks, abhishekk) and I have draft patches locally to do the same for Cinder which I just need to finish. Regarding the initial question, @zigo you can add me to core for sqlalchemy- migrate but with the caveat that I won't be sticking around to maintain it long- term. This library is now a liability and project need to move off of it. The combo of Nova, Glance and Cinder patch series that I'm proposing should get us a long way there and will serve as a blueprint for other project, so there isn't really a reason this can't happen in any healthy project over the next cycle. Stephen
Stephen Finucane wrote:
[...] Regarding the initial question, @zigo you can add me to core for sqlalchemy- migrate but with the caveat that I won't be sticking around to maintain it long- term. This library is now a liability and project need to move off of it.
Thanks Stephen!
The combo of Nova, Glance and Cinder patch series that I'm proposing should get us a long way there and will serve as a blueprint for other project, so there isn't really a reason this can't happen in any healthy project over the next cycle.
Sounds like a good idea for a Xena community goal. Yes it's not user-facing, but paying back technical debt like this is also important. -- Thierry
On 2/24/21 2:42 PM, Stephen Finucane wrote:
Regarding the initial question, @zigo you can add me to core for sqlalchemy- migrate but with the caveat that I won't be sticking around to maintain it long- term.
Thanks, you're in! I also removed people I know have given-up interest in sqla-migrate (like Matt Riedmann, Chuck Short). Is Michael Still around, or should I remove him too?
This library is now a liability and project need to move off of it.
Yeah! Let's kill it.
The combo of Nova, Glance and Cinder patch series that I'm proposing should get us a long way there and will serve as a blueprint for other project, so there isn't really a reason this can't happen in any healthy project over the next cycle.
There's also Heat, that uses migrate, if I'm not mistaking. IMO that's also a core project. Appart from it, there's also: - Desginate - Senlin - Trove I don't think there's anyone else that still uses it, so that's 4 core projects, and 3 "less core" projects. I based my survey out of the Debian dependency list, hopefully that doesn't include a mistake. Cheers, Thomas Goirand (zigo)
On Sun, Feb 28, 2021 at 4:25 PM Thomas Goirand <thomas@goirand.fr> wrote:
The combo of Nova, Glance and Cinder patch series that I'm proposing should get us a long way there and will serve as a blueprint for other project, so there isn't really a reason this can't happen in any healthy project over the next cycle.
There's also Heat, that uses migrate, if I'm not mistaking. IMO that's also a core project.
Appart from it, there's also: - Desginate - Senlin - Trove
I don't think there's anyone else that still uses it, so that's 4 core projects, and 3 "less core" projects.
I based my survey out of the Debian dependency list, hopefully that doesn't include a mistake.
It is pretty close! See https://codesearch.opendev.org/?q=sqlalchemy-migrate&i=nope&files=requirements.txt&excludeFiles=&repos= I caught Keystone, Murano and oslo.db too. -yoctozepto
On 2021-02-28 16:46:13 +0100 (+0100), Radosław Piliszek wrote:
On Sun, Feb 28, 2021 at 4:25 PM Thomas Goirand <thomas@goirand.fr> wrote:
The combo of Nova, Glance and Cinder patch series that I'm proposing should get us a long way there and will serve as a blueprint for other project, so there isn't really a reason this can't happen in any healthy project over the next cycle.
There's also Heat, that uses migrate, if I'm not mistaking. IMO that's also a core project.
Appart from it, there's also: - Desginate - Senlin - Trove
I don't think there's anyone else that still uses it, so that's 4 core projects, and 3 "less core" projects.
I based my survey out of the Debian dependency list, hopefully that doesn't include a mistake.
It is pretty close!
I caught Keystone, Murano and oslo.db too.
You may want to double-check that they actually import it. Earlier in this discussion I performed a similar search just to get an idea for how widespread the problem is, and noticed that some projects still have cruft entries in their requirements.txt even though they don't actually use the library any longer. -- Jeremy Stanley
On Sun, Feb 28, 2021 at 4:51 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2021-02-28 16:46:13 +0100 (+0100), Radosław Piliszek wrote:
On Sun, Feb 28, 2021 at 4:25 PM Thomas Goirand <thomas@goirand.fr> wrote:
The combo of Nova, Glance and Cinder patch series that I'm proposing should get us a long way there and will serve as a blueprint for other project, so there isn't really a reason this can't happen in any healthy project over the next cycle.
There's also Heat, that uses migrate, if I'm not mistaking. IMO that's also a core project.
Appart from it, there's also: - Desginate - Senlin - Trove
I don't think there's anyone else that still uses it, so that's 4 core projects, and 3 "less core" projects.
I based my survey out of the Debian dependency list, hopefully that doesn't include a mistake.
It is pretty close!
I caught Keystone, Murano and oslo.db too.
You may want to double-check that they actually import it. Earlier in this discussion I performed a similar search just to get an idea for how widespread the problem is, and noticed that some projects still have cruft entries in their requirements.txt even though they don't actually use the library any longer.
Wise suggestion. I do not see, however, the search you mention. Anyhow, I made the following two queries: https://codesearch.opendev.org/?q=%5Efrom%5Cs%2B%5Cbmigrate%5Cb&i=nope&files=%5C.py%24&excludeFiles=&repos= https://codesearch.opendev.org/?q=%5Eimport%5Cs%2B%5Cbmigrate%5Cb&i=nope&files=%5C.py%24&excludeFiles=&repos= So Murano was a false positive. But Keystone and oslo.db stay. On the other hand, there are a few others as well which have not had it declared (possibly due to oslo.db pulling it in). I saw: - Masakari* - Freezer * This is the point where I ask for guidelines of successful migration to alembic. PS: Apart from OpenStack, it seems StarlingX uses it. -yoctozepto
On 2021-02-28 17:23:13 +0100 (+0100), Radosław Piliszek wrote:
On Sun, Feb 28, 2021 at 4:51 PM Jeremy Stanley <fungi@yuggoth.org> wrote: [...]
Earlier in this discussion I performed a similar search just to get an idea for how widespread the problem is, and noticed that some projects still have cruft entries in their requirements.txt even though they don't actually use the library any longer.
Wise suggestion. I do not see, however, the search you mention. [...]
Yep, sorry that was confusing. What I was trying to say is that earlier in this discussion I got curious as to how widespread the problem was, so I did some cursory searches for myself. I did not actually report the results in the discussion, as it was not a thorough analysis and I did not feel it would have added much at the time. -- Jeremy Stanley
On 2/28/21 4:46 PM, Radosław Piliszek wrote:
On Sun, Feb 28, 2021 at 4:25 PM Thomas Goirand <thomas@goirand.fr> wrote:
The combo of Nova, Glance and Cinder patch series that I'm proposing should get us a long way there and will serve as a blueprint for other project, so there isn't really a reason this can't happen in any healthy project over the next cycle.
There's also Heat, that uses migrate, if I'm not mistaking. IMO that's also a core project.
Appart from it, there's also: - Desginate - Senlin - Trove
I don't think there's anyone else that still uses it, so that's 4 core projects, and 3 "less core" projects.
I based my survey out of the Debian dependency list, hopefully that doesn't include a mistake.
It is pretty close!
I caught Keystone, Murano and oslo.db too.
-yoctozepto
You are right for Keystone. Though... Yes, Murano has a dependency on it, however, as much as I can see, it still uses Alembic. I'm not sure why it still needs migrate. Maybe because it transitionned to Alembic and still has it in tests? Cheers, Thomas Goirand (zigo)
On Sun, Feb 28, 2021 at 5:23 PM Thomas Goirand <zigo@debian.org> wrote:
You are right for Keystone. Though...
Yes, Murano has a dependency on it, however, as much as I can see, it still uses Alembic. I'm not sure why it still needs migrate. Maybe because it transitionned to Alembic and still has it in tests?
Yes, it's probably just some old cruft there, please see my other message. You posted right after my amendment. -yoctozepto
On 2/28/21 5:59 PM, Radosław Piliszek wrote:
On Sun, Feb 28, 2021 at 5:23 PM Thomas Goirand <zigo@debian.org> wrote:
You are right for Keystone. Though...
Yes, Murano has a dependency on it, however, as much as I can see, it still uses Alembic. I'm not sure why it still needs migrate. Maybe because it transitionned to Alembic and still has it in tests?
Yes, it's probably just some old cruft there, please see my other message. You posted right after my amendment.
-yoctozep
So, to sum-up, we have still to fix: - Keystone - Nova - Glance - Cinder - Heat - Designate - Senlin - Trove - Murano (only cruft) That's a long enough list that probably, we will need SQLAM for at least one more year (and hopefully, not another decade...). Cheers, Thomas Goirand (zigo)
participants (10)
-
Ben Nemec
-
Jay Bryant
-
Jeremy Stanley
-
Radosław Piliszek
-
Sean McGinnis
-
Sean Mooney
-
Stephen Finucane
-
Thierry Carrez
-
Thomas Goirand
-
Thomas Goirand