[all] SLURP support survey
Hi,
The Kolla project is planning to support SLURP upgrades [1] in the Caracal release cycle, allowing users to upgrade from Antelope to Caracal. There are two areas in which I'd like to request information from the various project teams.
# 1. SLURP support
We understand that for various reasons there can be a gap between TC resolutions and implementation across projects. Using some rudimentary grepping through various codebases I found that many projects have added the grenade-skip-level job (or a variation of it) to their CI pipelines, and a few mention SLURP in their release notes. There's little mention of it in documentation at this stage.
AFAICT the following projects have SLURP upgrade jobs: Cinder, Glance, Ironic, Manila, Neutron, Nova, Octavia
While I could not find a SLURP job in these: Barbican, Bifrost, Blazar, Cloudkitty, Designate, Heat, Horizon, Keystone, Magnum, Placement
Question: Does your project explicitly support SLURP upgrades? Do you have or are you planning to add testing and documentation for SLURP upgrades?
# 2. Rolling SLURP upgrades
The TC resolution [1] states that rolling upgrades are optional for SLURP releases. This could affect the upgrade procedure required in Kolla Ansible. Nova mentions in its Antelope release notes that it supports rolling SLURP upgrades, but I could not find any information from other projects. It would be helpful if this was documented for each project.
Question: Does your project support or plan to support rolling upgrades? If changes to the upgrade procedure for SLURP are required, are they documented?
Thank you in advance for your responses. Apologies if I have missed any obvious source of information on this.
Regards, Mark
[1] https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adj...
Hi Mark,
Thank you for reaching out about this topic. I can answer for the Blazar and CloudKitty projects, in which I am involved.
Unfortunately these projects are not up to the standards of more mainstream OpenStack projects: neither of them are testing upgrades at all (no grenade job). There is no technical reason for not having them, just that the limited time from contributors was spent elsewhere.
Blazar should support rolling SLURP upgrades given there were no recent changes that would break compatibility or require a specific upgrade method. I think it should be the same for CloudKitty, but I will discuss it with the rest of the team next week.
Cheers, Pierre Riteau (priteau)
On Fri, 2 Feb 2024 at 12:12, Mark Goddard mark@stackhpc.com wrote:
Hi,
The Kolla project is planning to support SLURP upgrades [1] in the Caracal release cycle, allowing users to upgrade from Antelope to Caracal. There are two areas in which I'd like to request information from the various project teams.
# 1. SLURP support
We understand that for various reasons there can be a gap between TC resolutions and implementation across projects. Using some rudimentary grepping through various codebases I found that many projects have added the grenade-skip-level job (or a variation of it) to their CI pipelines, and a few mention SLURP in their release notes. There's little mention of it in documentation at this stage.
AFAICT the following projects have SLURP upgrade jobs: Cinder, Glance, Ironic, Manila, Neutron, Nova, Octavia
While I could not find a SLURP job in these: Barbican, Bifrost, Blazar, Cloudkitty, Designate, Heat, Horizon, Keystone, Magnum, Placement
Question: Does your project explicitly support SLURP upgrades? Do you have or are you planning to add testing and documentation for SLURP upgrades?
# 2. Rolling SLURP upgrades
The TC resolution [1] states that rolling upgrades are optional for SLURP releases. This could affect the upgrade procedure required in Kolla Ansible. Nova mentions in its Antelope release notes that it supports rolling SLURP upgrades, but I could not find any information from other projects. It would be helpful if this was documented for each project.
Question: Does your project support or plan to support rolling upgrades? If changes to the upgrade procedure for SLURP are required, are they documented?
Thank you in advance for your responses. Apologies if I have missed any obvious source of information on this.
Regards, Mark
[1] https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adj...
Hi,
On 2/2/24 12:12, Mark Goddard wrote:
Hi,
The Kolla project is planning to support SLURP upgrades [1] in the Caracal release cycle, allowing users to upgrade from Antelope to Caracal. There are two areas in which I'd like to request information from the various project teams.
# 1. SLURP support
We understand that for various reasons there can be a gap between TC resolutions and implementation across projects. Using some rudimentary grepping through various codebases I found that many projects have added the grenade-skip-level job (or a variation of it) to their CI pipelines, and a few mention SLURP in their release notes. There's little mention of it in documentation at this stage.
AFAICT the following projects have SLURP upgrade jobs: Cinder, Glance, Ironic, Manila, Neutron, Nova, Octavia
While I could not find a SLURP job in these: Barbican, Bifrost, Blazar, Cloudkitty, Designate, Heat, Horizon, Keystone, Magnum, Placement
Bifrost is a bit special in this list since it's a deployment tool itself. I haven't heard any conversations around supporting SLURP upgrades for it, nor am I particularly inclined to spend time on it.
Dmitry
Question: Does your project explicitly support SLURP upgrades? Do you have or are you planning to add testing and documentation for SLURP upgrades?
# 2. Rolling SLURP upgrades
The TC resolution [1] states that rolling upgrades are optional for SLURP releases. This could affect the upgrade procedure required in Kolla Ansible. Nova mentions in its Antelope release notes that it supports rolling SLURP upgrades, but I could not find any information from other projects. It would be helpful if this was documented for each project.
Question: Does your project support or plan to support rolling upgrades? If changes to the upgrade procedure for SLURP are required, are they documented?
Thank you in advance for your responses. Apologies if I have missed any obvious source of information on this.
Regards, Mark
[1] https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adj... https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adjustment.html
FWIW, in OpenStack-Ansible we have implemented SLURP upgrades as a training for Yoga -> Antelope (as first unofficial slurp upgrade), and results were already pretty satisfying.
There were no issues with upgrades for most used projects at very least, except a couple of bugs that happen regardless.
We also followed this upgrade path in our environments and it was very smooth, so I would expect that projects mainly do keep in mind slurp policy already.
Or we were just lucky with changes between Y -> 2023.1 :)
On Fri, Feb 2, 2024, 15:40 Dmitry Tantsur dtantsur@protonmail.com wrote:
Hi,
On 2/2/24 12:12, Mark Goddard wrote:
Hi,
The Kolla project is planning to support SLURP upgrades [1] in the Caracal release cycle, allowing users to upgrade from Antelope to Caracal. There are two areas in which I'd like to request information from the various project teams.
# 1. SLURP support
We understand that for various reasons there can be a gap between TC resolutions and implementation across projects. Using some rudimentary grepping through various codebases I found that many projects have added the grenade-skip-level job (or a variation of it) to their CI pipelines, and a few mention SLURP in their release notes. There's little mention of it in documentation at this stage.
AFAICT the following projects have SLURP upgrade jobs: Cinder, Glance, Ironic, Manila, Neutron, Nova, Octavia
While I could not find a SLURP job in these: Barbican, Bifrost, Blazar, Cloudkitty, Designate, Heat, Horizon, Keystone, Magnum, Placement
Bifrost is a bit special in this list since it's a deployment tool itself. I haven't heard any conversations around supporting SLURP upgrades for it, nor am I particularly inclined to spend time on it.
Dmitry
Question: Does your project explicitly support SLURP upgrades? Do you have or are you planning to add testing and documentation for SLURP upgrades?
# 2. Rolling SLURP upgrades
The TC resolution [1] states that rolling upgrades are optional for SLURP releases. This could affect the upgrade procedure required in Kolla Ansible. Nova mentions in its Antelope release notes that it supports rolling SLURP upgrades, but I could not find any information from other projects. It would be helpful if this was documented for each project.
Question: Does your project support or plan to support rolling upgrades? If changes to the upgrade procedure for SLURP are required, are they documented?
Thank you in advance for your responses. Apologies if I have missed any obvious source of information on this.
Regards, Mark
[1]
https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adj... < https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adj...
Am Freitag, dem 02.02.2024 um 17:08 +0100 schrieb Dmitriy Rabotyagov:
There were no issues with upgrades for most used projects at very least, except a couple of bugs that happen regardless.
I find these kinds of statements very contradicting.
were there "no issues" or "a couple of bugs"? These are really distinct things with a meaning. If there are bugs, that's an issue for me I would like to fix.
Do you happen to have a list of bug reports from these upgrades? Are all of these resolved?
Thanks
These bugs would be there even for Zed -> 2023.1 upgrade, so I think they are unrelated to the SLURP process.
While I don't have full list handy, but there're at least some of them that I can recall right away:
* https://bugs.launchpad.net/keystone/+bug/2030061 (some help with unit tests would be welcome here: https://review.opendev.org/c/openstack/keystone/+/893737) * https://bugs.launchpad.net/keystone/+bug/2028809 * https://bugs.launchpad.net/keystone/+bug/2029134 * https://bugs.launchpad.net/nova/+bug/2040264
So while the statement indeed sounds contradictory, in fact it's not, as bugs above are more or less generic regressions.
пн, 5 февр. 2024 г. в 12:15, Sven Kieske kieske@osism.tech:
Am Freitag, dem 02.02.2024 um 17:08 +0100 schrieb Dmitriy Rabotyagov:
There were no issues with upgrades for most used projects at very least, except a couple of bugs that happen regardless.
I find these kinds of statements very contradicting.
were there "no issues" or "a couple of bugs"? These are really distinct things with a meaning. If there are bugs, that's an issue for me I would like to fix.
Do you happen to have a list of bug reports from these upgrades? Are all of these resolved?
Thanks
-- Sven Kieske Senior Cloud Engineer
Mail: kieske@osism.tech Web: https://osism.tech
OSISM GmbH Teckstraße 62 / 70190 Stuttgart / Deutschland
Geschäftsführer: Christian Berendt Unternehmenssitz: Stuttgart Amtsgericht: Stuttgart, HRB 756139
On 2/2/24 20:12, Mark Goddard wrote:
Hi,
The Kolla project is planning to support SLURP upgrades [1] in the Caracal release cycle, allowing users to upgrade from Antelope to Caracal. There are two areas in which I'd like to request information from the various project teams.
# 1. SLURP support
We understand that for various reasons there can be a gap between TC resolutions and implementation across projects. Using some rudimentary grepping through various codebases I found that many projects have added the grenade-skip-level job (or a variation of it) to their CI pipelines, and a few mention SLURP in their release notes. There's little mention of it in documentation at this stage.
AFAICT the following projects have SLURP upgrade jobs: Cinder, Glance, Ironic, Manila, Neutron, Nova, Octavia
While I could not find a SLURP job in these: Barbican, Bifrost, Blazar, Cloudkitty, Designate, Heat, Horizon, Keystone, Magnum, Placement
Question: Does your project explicitly support SLURP upgrades? Do you have or are you planning to add testing and documentation for SLURP upgrades?
For heat, we haven't made any incompatible db schema change or message queue changes so direct upgrade from 2023.1 to 2024.1 is supposed to work. I've proposed a change to add skip-level job and seems the job is now passing. I don't expect that any doc update may be needed specifically for SLURP. https://review.opendev.org/c/openstack/heat/+/907735
I'm not too sure if we have to mention explicitly about SLURP in individual projects. (AFAIK SLURP is not something a project can opt-in but should be a global policy, or do we have a mechanism to state that the project does/will not support SLURP ?)
For the other projects as much as I know ...
Skip level upgrade of keystone and placement are tested in CI of the other projects but probably adding the same skip level job to these two projects would make sense. https://review.opendev.org/c/openstack/keystone/+/907749 https://review.opendev.org/c/openstack/placement/+/907742
I also created a change to add skip-level jobs for designate. The jobs are passing now which is promising. https://review.opendev.org/c/openstack/designate/+/907741
It seems grenade job in brabican is currently broken (idk why it's kept non-voting now) so skip-level upgrade job is not passing either, so we may need someone from Barbican to look into these. https://review.opendev.org/c/openstack/barbican/+/907750
I'm not too sure how we may test upgrade scenario specific to horizon. Horizon itself has no local datastore and truly rely on the backend apis. One point I can think of is testing horizon N+2 and the other services N but I think we don't cover scenario of mixing versions.
I'm not very familiar with the other projects (Blazar, Cloudkitty and Magnum) but they may also be able to try adding skip-level job following the existing approaches in a few projects like octavia[1] or what I proposed above. [1] https://github.com/openstack/octavia/blob/master/zuul.d/jobs.yaml#L91
# 2. Rolling SLURP upgrades
The TC resolution [1] states that rolling upgrades are optional for SLURP releases. This could affect the upgrade procedure required in Kolla Ansible. Nova mentions in its Antelope release notes that it supports rolling SLURP upgrades, but I could not find any information from other projects. It would be helpful if this was documented for each project.
Question: Does your project support or plan to support rolling upgrades? If changes to the upgrade procedure for SLURP are required, are they documented?
Thank you in advance for your responses. Apologies if I have missed any obvious source of information on this.
Regards, Mark
[1] https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adj... https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adjustment.html
On Tue, 2024-02-06 at 00:48 +0900, Takashi Kajinami wrote:
On 2/2/24 20:12, Mark Goddard wrote:
Hi,
The Kolla project is planning to support SLURP upgrades [1] in the Caracal release cycle, allowing users to upgrade from Antelope to Caracal. There are two areas in which I'd like to request information from the various project teams.
# 1. SLURP support
We understand that for various reasons there can be a gap between TC resolutions and implementation across projects. Using some rudimentary grepping through various codebases I found that many projects have added the grenade-skip- level job (or a variation of it) to their CI pipelines, and a few mention SLURP in their release notes. There's little mention of it in documentation at this stage.
AFAICT the following projects have SLURP upgrade jobs: Cinder, Glance, Ironic, Manila, Neutron, Nova, Octavia
While I could not find a SLURP job in these: Barbican, Bifrost, Blazar, Cloudkitty, Designate, Heat, Horizon, Keystone, Magnum, Placement
Question: Does your project explicitly support SLURP upgrades? Do you have or are you planning to add testing and documentation for SLURP upgrades?
For heat, we haven't made any incompatible db schema change or message queue changes so direct upgrade from 2023.1 to 2024.1 is supposed to work. I've proposed a change to add skip-level job and seems the job is now passing. I don't expect that any doc update may be needed specifically for SLURP. https://review.opendev.org/c/openstack/heat/+/907735
I'm not too sure if we have to mention explicitly about SLURP in individual projects. (AFAIK SLURP is not something a project can opt-in but should be a global policy, or do we have a mechanism to state that the project does/will not support SLURP ?)
For the other projects as much as I know ...
Skip level upgrade of keystone and placement are tested in CI of the other projects but probably adding the same skip level job to these two projects would make sense. https://review.opendev.org/c/openstack/keystone/+/907749 https://review.opendev.org/c/openstack/placement/+/907742
placement does nto really need a skip level upgrade job for oen it is tested by nova's implicly but placment and keystone are both just rest apis in front fo a db
we do not supprot mixign diffent version of the wsgi applicatiosn pointing at the same db concurrently so for both project the slurp process is effectivly just a db migration folowed by upgrading the packages and restarting the app.
we can add jobs for them but placment offically supprot slurp upgrades even without the skip level job.
I also created a change to add skip-level jobs for designate. The jobs are passing now which is promising. https://review.opendev.org/c/openstack/designate/+/907741
It seems grenade job in brabican is currently broken (idk why it's kept non-voting now) so skip-level upgrade job is not passing either, so we may need someone from Barbican to look into these. https://review.opendev.org/c/openstack/barbican/+/907750
I'm not too sure how we may test upgrade scenario specific to horizon. Horizon itself has no local datastore and truly rely on the backend apis. One point I can think of is testing horizon N+2 and the other services N but I think we don't cover scenario of mixing versions.
I'm not very familiar with the other projects (Blazar, Cloudkitty and Magnum) but they may also be able to try adding skip-level job following the existing approaches in a few projects like octavia[1] or what I proposed above. [1] https://github.com/openstack/octavia/blob/master/zuul.d/jobs.yaml#L91
# 2. Rolling SLURP upgrades
The TC resolution [1] states that rolling upgrades are optional for SLURP releases. This could affect the upgrade procedure required in Kolla Ansible. Nova mentions in its Antelope release notes that it supports rolling SLURP upgrades, but I could not find any information from other projects. It would be helpful if this was documented for each project.
Question: Does your project support or plan to support rolling upgrades? If changes to the upgrade procedure for SLURP are required, are they documented?
Thank you in advance for your responses. Apologies if I have missed any obvious source of information on this.
Regards, Mark
[1] https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adj...<https://governance.openstack.org/tc/resolutions/20220210-release-cadence-adj...
participants (7)
-
Dmitriy Rabotyagov
-
Dmitry Tantsur
-
Mark Goddard
-
Pierre Riteau
-
smooney＠redhat.com
-
Sven Kieske
-
Takashi Kajinami