Re: [openstack-dev] [all][qa] Migrating devstack jobs to Bionic (Ubuntu LTS 18.04)
Hi All, Let's go with approach 1 means migrating the Devstack and Tempest base jobs to Bionic. This will move most of the jobs to Bionic. We have two patches up which move all Devstack and Tempest jobs to Bionic and it's working fine. 1. All DevStack jobs to Bionic - https://review.openstack.org/#/c/610977/ - This will move devstack-minimal, devstack, devstack-ipv6, devstack-multinode jobs to bionic only for master which means it will be stein onwards. All these jobs will use xenial till stable/rocky. 2. All Tempest base jobs (except stable branches job running on master) to Bionic - https://review.openstack.org/#/c/618169/ - This will move devstack-tempest, tempest-all, devstack-tempest-ipv6, tempest-full, tempest-full-py3, tempest-multinode-full, tempest-slow jobs to bionic. Note- Even Tempest is branchless and these tempest jobs have been moved to Bionic, they will still use xenial for all stable branches(till stable/rocky) testing. with zuulv3 magic and devstack base jobs nodeset for stable branch (xenial) and master (stein onwards -bionic) will take care of that. Tested on [1] and working fine. Thanks corvus and clarkb for guiding to this optimized way. 3. Legacy jobs are not migrated to bionic. They should get migrated to Bionic while they are moved to zuulv3 native. So if your projects have many legacy jobs then, they will still run on xenial. Any job inherits from those base jobs will behave the same way (running on bionic from stein onwards and xenial till stable/rocky). I am writing the plan and next action item to complete this migration activity: 1 Project teams: need to test their jobs 1. which are inherited from devstack/tempest base jobs and should pass as it is 2. Any zuulv3 jobs not using devstack/tempest base job required to migrate to use bionic (Devstack patch provide the bionic nodeset) and test it. Start writing the results on etherpad[2] 2 QA team will merge the above patches by 10th Dec so that we can find and fix any issues as early and to avoid the same during release time. Let's finish the pre-testing till 10th Dec and then merge the bionic migration patches. [1] https://review.openstack.org/#/c/618181/ https://review.openstack.org/#/c/618176/ [2] https://etherpad.openstack.org/p/devstack-bionic -gmann ---- On Wed, 07 Nov 2018 08:45:45 +0900 Doug Hellmann <doug@doughellmann.com> wrote ----
Ghanshyam Mann <gmann@ghanshyammann.com> writes:
---- On Wed, 07 Nov 2018 06:51:32 +0900 Slawomir Kaplonski <skaplons@redhat.com> wrote ----
Hi,
Wiadomość napisana przez Jeremy Stanley <fungi@yuggoth.org> w dniu 06.11.2018, o godz. 22:25:
On 2018-11-06 22:05:49 +0100 (+0100), Slawek Kaplonski wrote: [...]
also add jobs like "devstack-xenial" and "tempest-full-xenial" which projects can use still for some time if their job on Bionic would be broken now? [...]
That opens the door to piecemeal migration, which (as we similarly saw during the Trusty to Xenial switch) will inevitably lead to projects who no longer gate on Xenial being unable to integration test against projects who don't yet support Bionic. At the same time, projects which have switched to Bionic will start merging changes which only work on Bionic without realizing it, so that projects which test on Xenial can't use them. In short, you'll be broken either way. On top of that, you can end up with projects that don't get around to switching completely before release comes, and then they're stuck having to manage a test platform transition on a stable branch.
I understand Your point here but will option 2) from first email lead to the same issues then?
seems so. approach 1 is less risky for such integrated testing issues and requires less work. In approach 1, we can coordinate the base job migration with project side testing with bionic.
-gmann
I like the approach of updating the devstack jobs to move everything to Bionic at one time because it sounds like it presents less risk of us ending up with something that looks like it works together but doesn't actually because it's tested on a different platform, as well as being less likely to cause us to have to do major porting work in stable branches after the release.
We'll need to take the same approach when updating the version of Python 3 used inside of devstack.
Doug
Reminder to test your project specific jobs if those are dependent on Devstack or Tempest base jobs and keep adding the results on etherpad- https://etherpad.openstack.org/p/devstack-bionic We will merge the Devstack and Tempest base job on Bionic on 10th Dec 2018. -gmann ---- On Thu, 22 Nov 2018 15:26:52 +0900 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hi All,
Let's go with approach 1 means migrating the Devstack and Tempest base jobs to Bionic. This will move most of the jobs to Bionic.
We have two patches up which move all Devstack and Tempest jobs to Bionic and it's working fine.
1. All DevStack jobs to Bionic - https://review.openstack.org/#/c/610977/ - This will move devstack-minimal, devstack, devstack-ipv6, devstack-multinode jobs to bionic only for master which means it will be stein onwards. All these jobs will use xenial till stable/rocky.
2. All Tempest base jobs (except stable branches job running on master) to Bionic - https://review.openstack.org/#/c/618169/ - This will move devstack-tempest, tempest-all, devstack-tempest-ipv6, tempest-full, tempest-full-py3, tempest-multinode-full, tempest-slow jobs to bionic. Note- Even Tempest is branchless and these tempest jobs have been moved to Bionic, they will still use xenial for all stable branches(till stable/rocky) testing. with zuulv3 magic and devstack base jobs nodeset for stable branch (xenial) and master (stein onwards -bionic) will take care of that. Tested on [1] and working fine. Thanks corvus and clarkb for guiding to this optimized way.
3. Legacy jobs are not migrated to bionic. They should get migrated to Bionic while they are moved to zuulv3 native. So if your projects have many legacy jobs then, they will still run on xenial.
Any job inherits from those base jobs will behave the same way (running on bionic from stein onwards and xenial till stable/rocky).
I am writing the plan and next action item to complete this migration activity:
1 Project teams: need to test their jobs 1. which are inherited from devstack/tempest base jobs and should pass as it is 2. Any zuulv3 jobs not using devstack/tempest base job required to migrate to use bionic (Devstack patch provide the bionic nodeset) and test it. Start writing the results on etherpad[2]
2 QA team will merge the above patches by 10th Dec so that we can find and fix any issues as early and to avoid the same during release time.
Let's finish the pre-testing till 10th Dec and then merge the bionic migration patches.
[1] https://review.openstack.org/#/c/618181/ https://review.openstack.org/#/c/618176/ [2] https://etherpad.openstack.org/p/devstack-bionic
-gmann
---- On Wed, 07 Nov 2018 08:45:45 +0900 Doug Hellmann <doug@doughellmann.com> wrote ----
Ghanshyam Mann <gmann@ghanshyammann.com> writes:
---- On Wed, 07 Nov 2018 06:51:32 +0900 Slawomir Kaplonski <skaplons@redhat.com> wrote ----
Hi,
Wiadomość napisana przez Jeremy Stanley <fungi@yuggoth.org> w dniu 06.11.2018, o godz. 22:25:
On 2018-11-06 22:05:49 +0100 (+0100), Slawek Kaplonski wrote: [...]
also add jobs like "devstack-xenial" and "tempest-full-xenial" which projects can use still for some time if their job on Bionic would be broken now? [...]
That opens the door to piecemeal migration, which (as we similarly saw during the Trusty to Xenial switch) will inevitably lead to projects who no longer gate on Xenial being unable to integration test against projects who don't yet support Bionic. At the same time, projects which have switched to Bionic will start merging changes which only work on Bionic without realizing it, so that projects which test on Xenial can't use them. In short, you'll be broken either way. On top of that, you can end up with projects that don't get around to switching completely before release comes, and then they're stuck having to manage a test platform transition on a stable branch.
I understand Your point here but will option 2) from first email lead to the same issues then?
seems so. approach 1 is less risky for such integrated testing issues and requires less work. In approach 1, we can coordinate the base job migration with project side testing with bionic.
-gmann
I like the approach of updating the devstack jobs to move everything to Bionic at one time because it sounds like it presents less risk of us ending up with something that looks like it works together but doesn't actually because it's tested on a different platform, as well as being less likely to cause us to have to do major porting work in stable branches after the release.
We'll need to take the same approach when updating the version of Python 3 used inside of devstack.
Doug
On Wednesday, 5 December 2018 13:02:08 CET Ghanshyam Mann wrote:
Reminder to test your project specific jobs if those are dependent on Devstack or Tempest base jobs and keep adding the results on etherpad- https://etherpad.openstack.org/p/devstack-bionic
We will merge the Devstack and Tempest base job on Bionic on 10th Dec 2018.
I can't test it right now using the gates (so I can't really report this on the etherpad), but a quick local test shows that devstack-plugin-ceph shows does not seem to support bionic. I may try to prepare a test job later if no one beats me at it. Ciao -- Luigi
On Wednesday, 5 December 2018 14:19:52 CET Luigi Toscano wrote:
On Wednesday, 5 December 2018 13:02:08 CET Ghanshyam Mann wrote:
Reminder to test your project specific jobs if those are dependent on Devstack or Tempest base jobs and keep adding the results on etherpad- https://etherpad.openstack.org/p/devstack-bionic
We will merge the Devstack and Tempest base job on Bionic on 10th Dec 2018.
I can't test it right now using the gates (so I can't really report this on the etherpad), but a quick local test shows that devstack-plugin-ceph shows does not seem to support bionic. I may try to prepare a test job later if no one beats me at it.
Erp, sorry, I didn't notice https://review.openstack.org/#/c/611594/ - I confirm that it makes devstack-plugin-ceph compatible with bionic, so please merge it :) Ciao -- Luigi
---- On Wed, 05 Dec 2018 22:29:10 +0900 Luigi Toscano <ltoscano@redhat.com> wrote ----
On Wednesday, 5 December 2018 14:19:52 CET Luigi Toscano wrote:
On Wednesday, 5 December 2018 13:02:08 CET Ghanshyam Mann wrote:
Reminder to test your project specific jobs if those are dependent on Devstack or Tempest base jobs and keep adding the results on etherpad- https://etherpad.openstack.org/p/devstack-bionic
We will merge the Devstack and Tempest base job on Bionic on 10th Dec 2018.
I can't test it right now using the gates (so I can't really report this on the etherpad), but a quick local test shows that devstack-plugin-ceph shows does not seem to support bionic. I may try to prepare a test job later if no one beats me at it.
Erp, sorry, I didn't notice https://review.openstack.org/#/c/611594/ - I confirm that it makes devstack-plugin-ceph compatible with bionic, so please merge it :)
Yeah, frickler had the fix up and now it is merged. Thanks. -gmann
Ciao -- Luigi
---- On Wed, 05 Dec 2018 21:02:08 +0900 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ---- Devstack and Tempest patch to move base job to Bionic are merged now. tempest-full, tempest-slow, etc + all jobs derived from tempest or devstack base jobs are running on bionic now. devstack provide bionic nodeset now, any zuulv3 native job running on xenail (zuulv3 jobs not derived from devstack/tempest base jobs) can be moved to bionic using those nodeset. Note- all the legacy jobs are still on xenial and they will be move to Bionic during migrating them to zullv3 native. -gmann
Reminder to test your project specific jobs if those are dependent on Devstack or Tempest base jobs and keep adding the results on etherpad- https://etherpad.openstack.org/p/devstack-bionic
We will merge the Devstack and Tempest base job on Bionic on 10th Dec 2018.
-gmann
---- On Thu, 22 Nov 2018 15:26:52 +0900 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hi All,
Let's go with approach 1 means migrating the Devstack and Tempest base jobs to Bionic. This will move most of the jobs to Bionic.
We have two patches up which move all Devstack and Tempest jobs to Bionic and it's working fine.
1. All DevStack jobs to Bionic - https://review.openstack.org/#/c/610977/ - This will move devstack-minimal, devstack, devstack-ipv6, devstack-multinode jobs to bionic only for master which means it will be stein onwards. All these jobs will use xenial till stable/rocky.
2. All Tempest base jobs (except stable branches job running on master) to Bionic - https://review.openstack.org/#/c/618169/ - This will move devstack-tempest, tempest-all, devstack-tempest-ipv6, tempest-full, tempest-full-py3, tempest-multinode-full, tempest-slow jobs to bionic. Note- Even Tempest is branchless and these tempest jobs have been moved to Bionic, they will still use xenial for all stable branches(till stable/rocky) testing. with zuulv3 magic and devstack base jobs nodeset for stable branch (xenial) and master (stein onwards -bionic) will take care of that. Tested on [1] and working fine. Thanks corvus and clarkb for guiding to this optimized way.
3. Legacy jobs are not migrated to bionic. They should get migrated to Bionic while they are moved to zuulv3 native. So if your projects have many legacy jobs then, they will still run on xenial.
Any job inherits from those base jobs will behave the same way (running on bionic from stein onwards and xenial till stable/rocky).
I am writing the plan and next action item to complete this migration activity:
1 Project teams: need to test their jobs 1. which are inherited from devstack/tempest base jobs and should pass as it is 2. Any zuulv3 jobs not using devstack/tempest base job required to migrate to use bionic (Devstack patch provide the bionic nodeset) and test it. Start writing the results on etherpad[2]
2 QA team will merge the above patches by 10th Dec so that we can find and fix any issues as early and to avoid the same during release time.
Let's finish the pre-testing till 10th Dec and then merge the bionic migration patches.
[1] https://review.openstack.org/#/c/618181/ https://review.openstack.org/#/c/618176/ [2] https://etherpad.openstack.org/p/devstack-bionic
-gmann
---- On Wed, 07 Nov 2018 08:45:45 +0900 Doug Hellmann <doug@doughellmann.com> wrote ----
Ghanshyam Mann <gmann@ghanshyammann.com> writes:
---- On Wed, 07 Nov 2018 06:51:32 +0900 Slawomir Kaplonski <skaplons@redhat.com> wrote ----
Hi,
Wiadomość napisana przez Jeremy Stanley <fungi@yuggoth.org> w dniu 06.11.2018, o godz. 22:25:
On 2018-11-06 22:05:49 +0100 (+0100), Slawek Kaplonski wrote: [...] > also add jobs like "devstack-xenial" and "tempest-full-xenial" > which projects can use still for some time if their job on Bionic > would be broken now? [...]
That opens the door to piecemeal migration, which (as we similarly saw during the Trusty to Xenial switch) will inevitably lead to projects who no longer gate on Xenial being unable to integration test against projects who don't yet support Bionic. At the same time, projects which have switched to Bionic will start merging changes which only work on Bionic without realizing it, so that projects which test on Xenial can't use them. In short, you'll be broken either way. On top of that, you can end up with projects that don't get around to switching completely before release comes, and then they're stuck having to manage a test platform transition on a stable branch.
I understand Your point here but will option 2) from first email lead to the same issues then?
seems so. approach 1 is less risky for such integrated testing issues and requires less work. In approach 1, we can coordinate the base job migration with project side testing with bionic.
-gmann
I like the approach of updating the devstack jobs to move everything to Bionic at one time because it sounds like it presents less risk of us ending up with something that looks like it works together but doesn't actually because it's tested on a different platform, as well as being less likely to cause us to have to do major porting work in stable branches after the release.
We'll need to take the same approach when updating the version of Python 3 used inside of devstack.
Doug
Ghanshyam Mann <gmann@ghanshyammann.com> writes:
---- On Wed, 05 Dec 2018 21:02:08 +0900 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Devstack and Tempest patch to move base job to Bionic are merged now. tempest-full, tempest-slow, etc + all jobs derived from tempest or devstack base jobs are running on bionic now.
devstack provide bionic nodeset now, any zuulv3 native job running on xenail (zuulv3 jobs not derived from devstack/tempest base jobs) can be moved to bionic using those nodeset.
Note- all the legacy jobs are still on xenial and they will be move to Bionic during migrating them to zullv3 native.
-gmann
Excellent! Thanks to you and the QA team for continuing to drive this migration. -- Doug
On 12/12/2018 6:21 AM, Ghanshyam Mann wrote:
Devstack and Tempest patch to move base job to Bionic are merged now. tempest-full, tempest-slow, etc + all jobs derived from tempest or devstack base jobs are running on bionic now.
Really? That's not what I'm seeing here: http://logs.openstack.org/77/610977/5/check/tempest-full-py3/e6c244a/zuul-in... Which is a job from that devstack patch to move to bionic. -- Thanks, Matt
On 12/12/2018 10:51 AM, Matt Riedemann wrote:
On 12/12/2018 6:21 AM, Ghanshyam Mann wrote:
Devstack and Tempest patch to move base job to Bionic are merged now. tempest-full, tempest-slow, etc + all jobs derived from tempest or devstack base jobs are running on bionic now.
Really? That's not what I'm seeing here:
http://logs.openstack.org/77/610977/5/check/tempest-full-py3/e6c244a/zuul-in...
Which is a job from that devstack patch to move to bionic.
Sorry for the noise, Clark pointed out the magic is turned on in the tempest change: https://review.openstack.org/#/c/618169/ -- Thanks, Matt
participants (4)
-
Doug Hellmann
-
Ghanshyam Mann
-
Luigi Toscano
-
Matt Riedemann