[all][tc][goals] Migrate CI/CD jobs to new Ubuntu LTS Focal: Week R-4 Update
Hello Everyone, Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan. Tracking: https://storyboard.openstack.org/#!/story/2007865 Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan. * Part1: Migrating tox base job tomorrow (8th Sept): ** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ . **There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing. * Part2: Migrating devstack/tempest base job on 10th sept: * We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept. ** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003 Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...) * ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open * ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe... Bugs Report: ========== 1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration. 2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003 3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now. 4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility. 5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging. What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack. 1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest: Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/ Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/ Example: https://review.opendev.org/#/c/738126/ 2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/ 3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2 4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches. Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches. References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865 [1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032 -gmann
On 07-09-20 09:29:40, Ghanshyam Mann wrote:
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration.
FWIW this looks like a QEMU 4.2 issue and I've raised the following bug: Second DEVICE_DELETED event missing during virtio-blk disk device detach https://bugs.launchpad.net/qemu/+bug/1894804 -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
* Part1: Migrating tox base job tomorrow (8th Sept):
I have checked it again and fixed many repos that are up for review and merge. Most python clients are already fixed or their fixes are up for merge so they can make it before the feature freeze on 10th. If any repo is broken then it will be pretty quick to fix by lower constraint bump (see the example under https://review.opendev.org/#/q/topic:migrate-to-focal) Even if any of the fixes miss the victoria release then those can be backported easily. I am opening the tox base jobs migration to merge: - All patches in this series https://review.opendev.org/#/c/738328/
* Part2: Migrating devstack/tempest base job on 10th sept:
We have three blocking open bugs here so I would like to discuss it in tomorrow's TC office hour also about how to proceed on this. 1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (https://bugs.launchpad.net/qemu/+bug/1894804) 2. Barbican: https://storyboard.openstack.org/#!/story/2007732 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121 -gmann ---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing.
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
* Part1: Migrating tox base job tomorrow (8th Sept):
I have checked it again and fixed many repos that are up for review and merge. Most python clients are already fixed or their fixes are up for merge so they can make it before the feature freeze on 10th. If any repo is broken then it will be pretty quick to fix by lower constraint bump (see the example under https://review.opendev.org/#/q/topic:migrate-to-focal)
Even if any of the fixes miss the victoria release then those can be backported easily. I am opening the tox base jobs migration to merge: - All patches in this series https://review.opendev.org/#/c/738328/
All these tox base jobs are merged now and running on Focal. If any of your repo is failing, please fix on priority or ping me on IRC if failure not clear. You can find most of the fixes for possible failure in this topic: - https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status...) -gmann
* Part2: Migrating devstack/tempest base job on 10th sept:
We have three blocking open bugs here so I would like to discuss it in tomorrow's TC office hour also about how to proceed on this.
1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (https://bugs.launchpad.net/qemu/+bug/1894804) 2. Barbican: https://storyboard.openstack.org/#!/story/2007732 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing.
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
Updates: Fixed a few more projects today which I found failing on Focal: - OpenStack SDKs repos : ready to merge - All remaining Oslo lib fixes: we are discussing FFE on these in separate ML thread. - Keystone: Fix is up, it should pass now. - Manila: Fix is up, it should pass gate. - Tacker: Ready to merge - neutron-dynamic-routing: Ready to merge - Cinder- it seems l-c job still failing. I will dig into it tomorrow or it will be appreciated if anyone can take a look before my morning. this is the patch -https://review.opendev.org/#/c/743080/ Note: all tox based jobs (Except py36/3.7) are running on Focal now so If any of you gate failing, feel free to ping me on #openstack-qa No more energy left for today, I will continue the remaining work tomorrow. -gmann ---- On Wed, 09 Sep 2020 14:05:17 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
* Part1: Migrating tox base job tomorrow (8th Sept):
I have checked it again and fixed many repos that are up for review and merge. Most python clients are already fixed or their fixes are up for merge so they can make it before the feature freeze on 10th. If any repo is broken then it will be pretty quick to fix by lower constraint bump (see the example under https://review.opendev.org/#/q/topic:migrate-to-focal)
Even if any of the fixes miss the victoria release then those can be backported easily. I am opening the tox base jobs migration to merge: - All patches in this series https://review.opendev.org/#/c/738328/
All these tox base jobs are merged now and running on Focal. If any of your repo is failing, please fix on priority or ping me on IRC if failure not clear. You can find most of the fixes for possible failure in this topic: - https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status...)
-gmann
* Part2: Migrating devstack/tempest base job on 10th sept:
We have three blocking open bugs here so I would like to discuss it in tomorrow's TC office hour also about how to proceed on this.
1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (https://bugs.launchpad.net/qemu/+bug/1894804) 2. Barbican: https://storyboard.openstack.org/#!/story/2007732 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing.
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
I've triaged this for Kolla and Kolla-Ansible too. Bifrost is also affected (but it's on Storyboard). -yoctozepto On Thu, Sep 10, 2020 at 5:42 AM Ghanshyam Mann <gmann@ghanshyammann.com> wrote:
Updates:
Fixed a few more projects today which I found failing on Focal:
- OpenStack SDKs repos : ready to merge - All remaining Oslo lib fixes: we are discussing FFE on these in separate ML thread. - Keystone: Fix is up, it should pass now. - Manila: Fix is up, it should pass gate. - Tacker: Ready to merge - neutron-dynamic-routing: Ready to merge - Cinder- it seems l-c job still failing. I will dig into it tomorrow or it will be appreciated if anyone can take a look before my morning. this is the patch -https://review.opendev.org/#/c/743080/
Note: all tox based jobs (Except py36/3.7) are running on Focal now so If any of you gate failing, feel free to ping me on #openstack-qa
No more energy left for today, I will continue the remaining work tomorrow.
-gmann
---- On Wed, 09 Sep 2020 14:05:17 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
* Part1: Migrating tox base job tomorrow (8th Sept):
I have checked it again and fixed many repos that are up for review and merge. Most python clients are already fixed or their fixes are up for merge so they can make it before the feature freeze on 10th. If any repo is broken then it will be pretty quick to fix by lower constraint bump (see the example under https://review.opendev.org/#/q/topic:migrate-to-focal)
Even if any of the fixes miss the victoria release then those can be backported easily. I am opening the tox base jobs migration to merge: - All patches in this series https://review.opendev.org/#/c/738328/
All these tox base jobs are merged now and running on Focal. If any of your repo is failing, please fix on priority or ping me on IRC if failure not clear. You can find most of the fixes for possible failure in this topic: - https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status...)
-gmann
* Part2: Migrating devstack/tempest base job on 10th sept:
We have three blocking open bugs here so I would like to discuss it in tomorrow's TC office hour also about how to proceed on this.
1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (https://bugs.launchpad.net/qemu/+bug/1894804) 2. Barbican: https://storyboard.openstack.org/#!/story/2007732 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing.
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
---- On Thu, 10 Sep 2020 02:18:02 -0500 Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote ----
I've triaged this for Kolla and Kolla-Ansible too.
Bifrost is also affected (but it's on Storyboard).
Thanks yoctozepto for fixing these. -gmann
-yoctozepto
On Thu, Sep 10, 2020 at 5:42 AM Ghanshyam Mann <gmann@ghanshyammann.com> wrote:
Updates:
Fixed a few more projects today which I found failing on Focal:
- OpenStack SDKs repos : ready to merge - All remaining Oslo lib fixes: we are discussing FFE on these in separate ML thread. - Keystone: Fix is up, it should pass now. - Manila: Fix is up, it should pass gate. - Tacker: Ready to merge - neutron-dynamic-routing: Ready to merge - Cinder- it seems l-c job still failing. I will dig into it tomorrow or it will be appreciated if anyone can take a look before my morning. this is the patch -https://review.opendev.org/#/c/743080/
Note: all tox based jobs (Except py36/3.7) are running on Focal now so If any of you gate failing, feel free to ping me on #openstack-qa
No more energy left for today, I will continue the remaining work tomorrow.
-gmann
---- On Wed, 09 Sep 2020 14:05:17 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
* Part1: Migrating tox base job tomorrow (8th Sept):
I have checked it again and fixed many repos that are up for review and merge. Most python clients are already fixed or their fixes are up for merge so they can make it before the feature freeze on 10th. If any repo is broken then it will be pretty quick to fix by lower constraint bump (see the example under https://review.opendev.org/#/q/topic:migrate-to-focal)
Even if any of the fixes miss the victoria release then those can be backported easily. I am opening the tox base jobs migration to merge: - All patches in this series https://review.opendev.org/#/c/738328/
All these tox base jobs are merged now and running on Focal. If any of your repo is failing, please fix on priority or ping me on IRC if failure not clear. You can find most of the fixes for possible failure in this topic: - https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status...)
-gmann
* Part2: Migrating devstack/tempest base job on 10th sept:
We have three blocking open bugs here so I would like to discuss it in tomorrow's TC office hour also about how to proceed on this.
1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (https://bugs.launchpad.net/qemu/+bug/1894804) 2. Barbican: https://storyboard.openstack.org/#!/story/2007732 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing.
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
Hi gmann, Sorry for that we've not merged your patch to Tacker because devstack on Focal fails in functional test. It seems gnocchi installation on Focal has some problems. Anyway, although this issue isn't fixed yet, we'll proceed to merge the patch immediately. Thanks, Yasufumi On 2020/09/10 12:32, Ghanshyam Mann wrote:
Updates:
Fixed a few more projects today which I found failing on Focal:
- OpenStack SDKs repos : ready to merge - All remaining Oslo lib fixes: we are discussing FFE on these in separate ML thread. - Keystone: Fix is up, it should pass now. - Manila: Fix is up, it should pass gate. - Tacker: Ready to merge - neutron-dynamic-routing: Ready to merge - Cinder- it seems l-c job still failing. I will dig into it tomorrow or it will be appreciated if anyone can take a look before my morning. this is the patch -https://review.opendev.org/#/c/743080/
Note: all tox based jobs (Except py36/3.7) are running on Focal now so If any of you gate failing, feel free to ping me on #openstack-qa
No more energy left for today, I will continue the remaining work tomorrow.
-gmann
---- On Wed, 09 Sep 2020 14:05:17 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
* Part1: Migrating tox base job tomorrow (8th Sept):
I have checked it again and fixed many repos that are up for review and merge. Most python clients are already fixed or their fixes are up for merge so they can make it before the feature freeze on 10th. If any repo is broken then it will be pretty quick to fix by lower constraint bump (see the example under https://review.opendev.org/#/q/topic:migrate-to-focal)
Even if any of the fixes miss the victoria release then those can be backported easily. I am opening the tox base jobs migration to merge: - All patches in this series https://review.opendev.org/#/c/738328/
All these tox base jobs are merged now and running on Focal. If any of your repo is failing, please fix on priority or ping me on IRC if failure not clear. You can find most of the fixes for possible failure in this topic: - https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status...)
-gmann
* Part2: Migrating devstack/tempest base job on 10th sept:
We have three blocking open bugs here so I would like to discuss it in tomorrow's TC office hour also about how to proceed on this.
1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (https://bugs.launchpad.net/qemu/+bug/1894804) 2. Barbican: https://storyboard.openstack.org/#!/story/2007732 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing.
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
---- On Thu, 10 Sep 2020 04:31:17 -0500 Yasufumi Ogawa <yasufum.o@gmail.com> wrote ----
Hi gmann,
Sorry for that we've not merged your patch to Tacker because devstack on Focal fails in functional test. It seems gnocchi installation on Focal has some problems.
Anyway, although this issue isn't fixed yet, we'll proceed to merge the patch immediately.
Thanks Yasufumi. I reported the gnoochi issue in the below storyboard and tried to reach out to the ceilometer team also but found not get any response. I will check what to do on this blocker. https://storyboard.openstack.org/#!/story/2008121 -gmann
Thanks, Yasufumi
On 2020/09/10 12:32, Ghanshyam Mann wrote:
Updates:
Fixed a few more projects today which I found failing on Focal:
- OpenStack SDKs repos : ready to merge - All remaining Oslo lib fixes: we are discussing FFE on these in separate ML thread. - Keystone: Fix is up, it should pass now. - Manila: Fix is up, it should pass gate. - Tacker: Ready to merge - neutron-dynamic-routing: Ready to merge - Cinder- it seems l-c job still failing. I will dig into it tomorrow or it will be appreciated if anyone can take a look before my morning. this is the patch -https://review.opendev.org/#/c/743080/
Note: all tox based jobs (Except py36/3.7) are running on Focal now so If any of you gate failing, feel free to ping me on #openstack-qa
No more energy left for today, I will continue the remaining work tomorrow.
-gmann
---- On Wed, 09 Sep 2020 14:05:17 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
* Part1: Migrating tox base job tomorrow (8th Sept):
I have checked it again and fixed many repos that are up for review and merge. Most python clients are already fixed or their fixes are up for merge so they can make it before the feature freeze on 10th. If any repo is broken then it will be pretty quick to fix by lower constraint bump (see the example under https://review.opendev.org/#/q/topic:migrate-to-focal)
Even if any of the fixes miss the victoria release then those can be backported easily. I am opening the tox base jobs migration to merge: - All patches in this series https://review.opendev.org/#/c/738328/
All these tox base jobs are merged now and running on Focal. If any of your repo is failing, please fix on priority or ping me on IRC if failure not clear. You can find most of the fixes for possible failure in this topic: - https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status...)
-gmann
* Part2: Migrating devstack/tempest base job on 10th sept:
We have three blocking open bugs here so I would like to discuss it in tomorrow's TC office hour also about how to proceed on this.
1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (https://bugs.launchpad.net/qemu/+bug/1894804) 2. Barbican: https://storyboard.openstack.org/#!/story/2007732 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing.
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
On 10/09/2020 15:43, Ghanshyam Mann wrote:
---- On Thu, 10 Sep 2020 04:31:17 -0500 Yasufumi Ogawa <yasufum.o@gmail.com> wrote ----
Hi gmann,
Sorry for that we've not merged your patch to Tacker because devstack on Focal fails in functional test. It seems gnocchi installation on Focal has some problems.
Anyway, although this issue isn't fixed yet, we'll proceed to merge the patch immediately.
Thanks Yasufumi.
I reported the gnoochi issue in the below storyboard and tried to reach out to the ceilometer team also but found not get any response. I will check what to do on this blocker.
So, how did you reach out, or who did you contact? Since gnocchi is a separate project outside of OpenStack, you should report these issues on https://github.com/gnocchixyz/gnocchi/issues. Especially, one should use the usual way to report issues for a project. Thank you for your patch for ceilometer, I did a review on it but did not get an answer to my question. Matthias
-gmann
Thanks, Yasufumi
On 2020/09/10 12:32, Ghanshyam Mann wrote:
Updates:
Fixed a few more projects today which I found failing on Focal:
- OpenStack SDKs repos : ready to merge - All remaining Oslo lib fixes: we are discussing FFE on these in separate ML thread. - Keystone: Fix is up, it should pass now. - Manila: Fix is up, it should pass gate. - Tacker: Ready to merge - neutron-dynamic-routing: Ready to merge - Cinder- it seems l-c job still failing. I will dig into it tomorrow or it will be appreciated if anyone can take a look before my morning. this is the patch -https://review.opendev.org/#/c/743080/
Note: all tox based jobs (Except py36/3.7) are running on Focal now so If any of you gate failing, feel free to ping me on #openstack-qa
No more energy left for today, I will continue the remaining work tomorrow.
-gmann
---- On Wed, 09 Sep 2020 14:05:17 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
* Part1: Migrating tox base job tomorrow (8th Sept):
I have checked it again and fixed many repos that are up for review and merge. Most python clients are already fixed or their fixes are up for merge so they can make it before the feature freeze on 10th. If any repo is broken then it will be pretty quick to fix by lower constraint bump (see the example under https://review.opendev.org/#/q/topic:migrate-to-focal)
Even if any of the fixes miss the victoria release then those can be backported easily. I am opening the tox base jobs migration to merge: - All patches in this series https://review.opendev.org/#/c/738328/
All these tox base jobs are merged now and running on Focal. If any of your repo is failing, please fix on priority or ping me on IRC if failure not clear. You can find most of the fixes for possible failure in this topic: - https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status...)
-gmann
* Part2: Migrating devstack/tempest base job on 10th sept:
We have three blocking open bugs here so I would like to discuss it in tomorrow's TC office hour also about how to proceed on this.
1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (https://bugs.launchpad.net/qemu/+bug/1894804) 2. Barbican: https://storyboard.openstack.org/#!/story/2007732 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing.
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
---- On Thu, 10 Sep 2020 15:13:45 -0500 Matthias Runge <mrunge@matthias-runge.de> wrote ----
On 10/09/2020 15:43, Ghanshyam Mann wrote:
---- On Thu, 10 Sep 2020 04:31:17 -0500 Yasufumi Ogawa <yasufum.o@gmail.com> wrote ----
Hi gmann,
Sorry for that we've not merged your patch to Tacker because devstack on Focal fails in functional test. It seems gnocchi installation on Focal has some problems.
Anyway, although this issue isn't fixed yet, we'll proceed to merge the patch immediately.
Thanks Yasufumi.
I reported the gnoochi issue in the below storyboard and tried to reach out to the ceilometer team also but found not get any response. I will check what to do on this blocker.
So, how did you reach out, or who did you contact?
Since gnocchi is a separate project outside of OpenStack, you should report these issues on https://github.com/gnocchixyz/gnocchi/issues. Especially, one should use the usual way to report issues for a project.
Thank you for your patch for ceilometer, I did a review on it but did not get an answer to my question.
Hi Matthias, I posted about this on the telemetry IRC channel. I have reported it on gnoochi github also - https://github.com/gnocchixyz/gnocchi/issues/1069 For the ceilometer patch, I updated it with lxml==4.2.3 which worked fine, hope it is ok now. -gmann
Matthias
-gmann
Thanks, Yasufumi
On 2020/09/10 12:32, Ghanshyam Mann wrote:
Updates:
Fixed a few more projects today which I found failing on Focal:
- OpenStack SDKs repos : ready to merge - All remaining Oslo lib fixes: we are discussing FFE on these in separate ML thread. - Keystone: Fix is up, it should pass now. - Manila: Fix is up, it should pass gate. - Tacker: Ready to merge - neutron-dynamic-routing: Ready to merge - Cinder- it seems l-c job still failing. I will dig into it tomorrow or it will be appreciated if anyone can take a look before my morning. this is the patch -https://review.opendev.org/#/c/743080/
Note: all tox based jobs (Except py36/3.7) are running on Focal now so If any of you gate failing, feel free to ping me on #openstack-qa
No more energy left for today, I will continue the remaining work tomorrow.
-gmann
---- On Wed, 09 Sep 2020 14:05:17 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
> * Part1: Migrating tox base job tomorrow (8th Sept):
I have checked it again and fixed many repos that are up for review and merge. Most python clients are already fixed or their fixes are up for merge so they can make it before the feature freeze on 10th. If any repo is broken then it will be pretty quick to fix by lower constraint bump (see the example under https://review.opendev.org/#/q/topic:migrate-to-focal)
Even if any of the fixes miss the victoria release then those can be backported easily. I am opening the tox base jobs migration to merge: - All patches in this series https://review.opendev.org/#/c/738328/
All these tox base jobs are merged now and running on Focal. If any of your repo is failing, please fix on priority or ping me on IRC if failure not clear. You can find most of the fixes for possible failure in this topic: - https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status...)
-gmann
> * Part2: Migrating devstack/tempest base job on 10th sept:
We have three blocking open bugs here so I would like to discuss it in tomorrow's TC office hour also about how to proceed on this.
1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (https://bugs.launchpad.net/qemu/+bug/1894804) 2. Barbican: https://storyboard.openstack.org/#!/story/2007732 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ---- > Hello Everyone, > > Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can > break the projects gate if not yet taken care of. Read below for the plan. > > Tracking: https://storyboard.openstack.org/#!/story/2007865 > > Progress: > ======= > * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add > unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate > which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below > plan. > > * Part1: Migrating tox base job tomorrow (8th Sept): > > ** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this > series (all base patches of this): https://review.opendev.org/#/c/738328/ . > > **There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. > Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing. > > * Part2: Migrating devstack/tempest base job on 10th sept: > > * We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept. > > ** Bug#1882521 > ** DB migration issues, > *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003 > > > Testing Till now: > ============ > * ~200 repos gate have been tested or fixed till now. > ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...) > > * ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your > project repos if I am late to fix them): > ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open > > * ~30repos fixes ready to merge: > ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe... > > > Bugs Report: > ========== > > 1. Bug#1882521. (IN-PROGRESS) > There is open bug for nova/cinder where three tempest tests are failing for > volume detach operation. There is no clear root cause found yet > -https://bugs.launchpad.net/cinder/+bug/1882521 > We have skipped the tests in tempest base patch to proceed with the other > projects testing but this is blocking things for the migration. > > 2. DB migration issues (IN-PROGRESS) > * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003 > > 3. We encountered the nodeset name conflict with x/tobiko. (FIXED) > nodeset conflict is resolved now and devstack provides all focal nodes now. > > 4. Bug#1886296. (IN-PROGRESS) > pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version > on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump > the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. > As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 > on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] > nd will release a new hacking version. After that project can move to new hacking and do not need > to maintain pyflakes version compatibility. > > 5. Bug#1886298. (IN-PROGRESS) > 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], > We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. > There are a few more issues[5] with lower-constraint jobs which I am debugging. > > > What work to be done on the project side: > ================================ > This goal is more of testing the jobs on focal and fixing bugs if any otherwise > migrate jobs by switching the nodeset to focal node sets defined in devstack. > > 1. Start a patch in your repo by making depends-on on either of below: > devstack base patch if you are using only devstack base jobs not tempest: > > Depends-on: https://review.opendev.org/#/c/731207/ > OR > tempest base patch if you are using the tempest base job (like devstack-tempest): > Depends-on: https://review.opendev.org/#/c/734700/ > > Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So > you can test the complete gate jobs(unit/functional/doc/integration) together. > This and its base patches - https://review.opendev.org/#/c/738328/ > > Example: https://review.opendev.org/#/c/738126/ > > 2. If none of your project jobs override the nodeset then above patch will be > testing patch(do not merge) otherwise change the nodeset to focal. > Example: https://review.opendev.org/#/c/737370/ > > 3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches > variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset > is overridden then devstack being branched and stable base job using bionic/xenial will take care of > this. > Example: https://review.opendev.org/#/c/744056/2 > > 4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need > updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On > so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in > this migration. > Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG > > Once we finish the testing on projects side and no failure then we will merge the devstack and tempest > base patches. > > > Important things to note: > =================== > * Do not forgot to add the story and task link to your patch so that we can track it smoothly. > * Use gerrit topic 'migrate-to-focal' > * Do not backport any of the patches. > > > References: > ========= > Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... > Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865 > > [1] https://github.com/PyCQA/pyflakes/issues/367 > [2] https://review.opendev.org/#/c/739315/ > [3] https://review.opendev.org/#/c/739334/ > [4] https://github.com/pallets/markupsafe/issues/116 > [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032 > > -gmann > >
On Wed, 2020-09-09 at 14:05 -0500, Ghanshyam Mann wrote:
---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann < gmann@ghanshyammann.com> wrote ----
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
* Part1: Migrating tox base job tomorrow (8th Sept):
I have checked it again and fixed many repos that are up for review and merge. Most python clients are already fixed or their fixes are up for merge so they can make it before the feature freeze on 10th. If any repo is broken then it will be pretty quick to fix by lower constraint bump (see the example under https://review.opendev.org/#/q/topic:migrate-to-focal)
Even if any of the fixes miss the victoria release then those can be backported easily. I am opening the tox base jobs migration to merge: - All patches in this series https://review.opendev.org/#/c/738328/
All these tox base jobs are merged now and running on Focal. If any of your repo is failing, please fix on priority or ping me on IRC if failure not clear. You can find most of the fixes for possible failure in this topic: - https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status...)
-gmann
We're in a bit of a pickle here. So with kuryr-kubernetes we aim to keep lower-constraints on the versions that can be found in CentOS/RHEL8 and seems like cffi 1.11.5 won't compile with Python 3.8. What should we do here? Is such assumption even possible given broader OpenStack assumptions?
* Part2: Migrating devstack/tempest base job on 10th sept:
We have three blocking open bugs here so I would like to discuss
it in tomorrow's TC office hour also about how to proceed on this.
1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (
2. Barbican: https://storyboard.openstack.org/#!/story/2007732 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann < gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is
series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower- constraints job specifically which I tried my best to fix as many as
Many are ready to merge also. Please merge or work on your
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we
will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side
https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. **
https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing
are in progress (If you would like to help, please check your
project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with
projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is
on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above
testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch ( https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so
https://bugs.launchpad.net/qemu/+bug/1894804) this possible. projects repo testing before that or fix on priority if failing. the other the default python version patch will be that we can track it smoothly.
* Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
I agree with Michał that it kind of breaks the purpose of lower-constraints. Supposedly lower-constraints should just be tested with the lowest supported python version? WDYT, Folks? (That said, lots of projects already made lower-constraints break on RDO due to these bumps.) -yoctozepto On Fri, Sep 11, 2020 at 2:02 PM Michał Dulko <mdulko@redhat.com> wrote:
On Wed, 2020-09-09 at 14:05 -0500, Ghanshyam Mann wrote:
---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann < gmann@ghanshyammann.com> wrote ----
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
* Part1: Migrating tox base job tomorrow (8th Sept):
I have checked it again and fixed many repos that are up for review and merge. Most python clients are already fixed or their fixes are up for merge so they can make it before the feature freeze on 10th. If any repo is broken then it will be pretty quick to fix by lower constraint bump (see the example under https://review.opendev.org/#/q/topic:migrate-to-focal)
Even if any of the fixes miss the victoria release then those can be backported easily. I am opening the tox base jobs migration to merge: - All patches in this series https://review.opendev.org/#/c/738328/
All these tox base jobs are merged now and running on Focal. If any of your repo is failing, please fix on priority or ping me on IRC if failure not clear. You can find most of the fixes for possible failure in this topic: - https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status...)
-gmann
We're in a bit of a pickle here. So with kuryr-kubernetes we aim to keep lower-constraints on the versions that can be found in CentOS/RHEL8 and seems like cffi 1.11.5 won't compile with Python 3.8. What should we do here? Is such assumption even possible given broader OpenStack assumptions?
* Part2: Migrating devstack/tempest base job on 10th sept:
We have three blocking open bugs here so I would like to discuss
it in tomorrow's TC office hour also about how to proceed on this.
1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (
2. Barbican: https://storyboard.openstack.org/#!/story/2007732 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann < gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is
series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower- constraints job specifically which I tried my best to fix as many as
Many are ready to merge also. Please merge or work on your
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we
will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side
https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. **
https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing
are in progress (If you would like to help, please check your
project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with
projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is
on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above
testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch ( https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so
https://bugs.launchpad.net/qemu/+bug/1894804) this possible. projects repo testing before that or fix on priority if failing. the other the default python version patch will be that we can track it smoothly.
* Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
---- On Fri, 11 Sep 2020 07:13:14 -0500 Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote ----
I agree with Michał that it kind of breaks the purpose of lower-constraints. Supposedly lower-constraints should just be tested with the lowest supported python version? WDYT, Folks?
(That said, lots of projects already made lower-constraints break on RDO due to these bumps.)
This is something we discussed yesterday, there are both way we can argue whether we should test l-c on lower supported python or available python on tested Disro which we do with Ubuntu Focal for all tox based jobs. And believe be lower constraints are not only python version things. But It is fine for projects say kuryr-kubernetes to keep running the l-c job on Bionic or centos I can provide patch for that. and for RHEL compatible version, it can be adjusted as I did in ceilometer for lxml but I am not sure if we can test all of them for RHEL also. - https://review.opendev.org/#/c/744612/ I am going to submit a Forum session to discuss on this topic so that we have an agreed way of testing in future, -gman
-yoctozepto
On Fri, Sep 11, 2020 at 2:02 PM Michał Dulko <mdulko@redhat.com> wrote:
On Wed, 2020-09-09 at 14:05 -0500, Ghanshyam Mann wrote:
---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann < gmann@ghanshyammann.com> wrote ----
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
* Part1: Migrating tox base job tomorrow (8th Sept):
I have checked it again and fixed many repos that are up for review and merge. Most python clients are already fixed or their fixes are up for merge so they can make it before the feature freeze on 10th. If any repo is broken then it will be pretty quick to fix by lower constraint bump (see the example under https://review.opendev.org/#/q/topic:migrate-to-focal)
Even if any of the fixes miss the victoria release then those can be backported easily. I am opening the tox base jobs migration to merge: - All patches in this series https://review.opendev.org/#/c/738328/
All these tox base jobs are merged now and running on Focal. If any of your repo is failing, please fix on priority or ping me on IRC if failure not clear. You can find most of the fixes for possible failure in this topic: - https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status...)
-gmann
We're in a bit of a pickle here. So with kuryr-kubernetes we aim to keep lower-constraints on the versions that can be found in CentOS/RHEL8 and seems like cffi 1.11.5 won't compile with Python 3.8. What should we do here? Is such assumption even possible given broader OpenStack assumptions?
* Part2: Migrating devstack/tempest base job on 10th sept:
We have three blocking open bugs here so I would like to discuss
it in tomorrow's TC office hour also about how to proceed on this.
1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (
2. Barbican: https://storyboard.openstack.org/#!/story/2007732 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann < gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is
series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower- constraints job specifically which I tried my best to fix as many as
Many are ready to merge also. Please merge or work on your
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we
will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side
https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. **
https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing
are in progress (If you would like to help, please check your
project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with
projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is
on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above
testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch ( https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so
https://bugs.launchpad.net/qemu/+bug/1894804) this possible. projects repo testing before that or fix on priority if failing. the other the default python version patch will be that we can track it smoothly.
* Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
On 2020-09-11 13:50:57 +0200 (+0200), Michał Dulko wrote: [...]
We're in a bit of a pickle here. So with kuryr-kubernetes we aim to keep lower-constraints on the versions that can be found in CentOS/RHEL8 and seems like cffi 1.11.5 won't compile with Python 3.8. What should we do here? Is such assumption even possible given broader OpenStack assumptions? [...]
To rehash and summarize my opinion from our lengthy IRC discussion in the #openstack-infra channel yesterday, I agree that it makes sense to run lower-constraints jobs on the lowest Python interpreter release listed in our tested runtimes for that cycle. However, you also have to take into account the platform(s) which cause that interpreter version to be included. Basically if we're going to standardize Python 3.6 jobs of any kind in Victoria those should be running on CentOS 8 (which is in our tested runtimes for Victoria and why we included Python 3.6 at all), *not* on Ubuntu Bionic (which is not included in our tested runtimes for Victoria). Otherwise we're stuck supporting ubuntu-bionic images for testing stable/victoria well into the future even though we didn't technically list it as a target platform. This gets more complicated though. We've, as a community overall, tried repeatedly to pretend we're not testing specific platforms, and instead just testing representative Python versions. Except we don't actually test with real Python versions we test with the patched Python interpreters shipped in distros. So our job names pretend they're Python version specific for the most part, when to actually meet the intent of our tested runtimes we really do need distro-specific jobs. We test Python 3.6 for stable/ussuri changes because Ubuntu Bionic is a tested runtime platform for Ussuri and that's the default Python 3 it ships, but Python 3.6 is listed in Victoria for the benefit of CentOS 8 not Ubuntu Bionic, so we need a different py36 job on each branch, running on different distros. -- Jeremy Stanley
---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Updates: After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today and discuss the integration testing switch tomorrow in TC office hours.
* Part1: Migrating tox base job tomorrow (8th Sept):
This is done and almost all the projects are fixed or at least fixes are up to merge. cinder and keystone l-c job on Focal also working fine and ready to merge. Few python clients have not yet merged the fixes so I have backported those to Victoria to fix master as well as Victoria gate. [...]
* Part2: Migrating devstack/tempest base job on 10th sept: We have three blocking open bugs here so I would like to discuss it in tomorrow's TC office hour also about how to proceed on this.
1. Nova: https://bugs.launchpad.net/nova/+bug/1882521 (https://bugs.launchpad.net/qemu/+bug/1894804) As of now, no work needed on Nova side and after QEMU fix, we will see if tests pass.
2. Barbican: https://storyboard.openstack.org/#!/story/2007732 The only blocker left for Focal migration.
3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121 This worked fine with mariadb - https://review.opendev.org/#/c/752294/
There are many fixes still not merged yet and their gate is also failing, I request to merge the fixes on priority: - https://review.opendev.org/#/q/topic:migrate-to-focal+status:open -gmann
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing.
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
Updates: * Ceilometer fix is ready to merge with +A - https://review.opendev.org/#/c/752294/ * Barbican fix is not yet merged but I think we should move on and switch the integration testing to Focal and keeping the barbican based job keep running on Bionic. * If you have any of the job failing on Focal and can not be fixed quickly then set the bionic nodeset for those to avoid gate block. Examples are below: - https://review.opendev.org/#/c/743079/4/.zuul.yaml@31 - https://review.opendev.org/#/c/743124/3/.zuul.d/base.yaml * I am planning to switch the devstck and tempest base job by tomorrow or the day after tomorrow, please take appropriate action in advance if your project is not tested or failing. Testing Status: =========== * ~270 repos gate have been tested green or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...) * ~28 repos are failing. Need immediate action mentioned above. ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open * ~18repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe... -gmann
-gmann
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing.
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
---- On Mon, 21 Sep 2020 17:37:21 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Updates:
* Ceilometer fix is ready to merge with +A - https://review.opendev.org/#/c/752294/
* Barbican fix is not yet merged but I think we should move on and switch the integration testing to Focal and keeping the barbican based job keep running on Bionic.
* If you have any of the job failing on Focal and can not be fixed quickly then set the bionic nodeset for those to avoid gate block. Examples are below: - https://review.opendev.org/#/c/743079/4/.zuul.yaml@31 - https://review.opendev.org/#/c/743124/3/.zuul.d/base.yaml
* I am planning to switch the devstck and tempest base job by tomorrow or the day after tomorrow, please take appropriate action in advance if your project is not tested or failing.
Devstack patch is merged which run devstack base jobs on Focal now. But Tempest patch is still in gate, so you might see tempest(-based) jobs failing on your gate, hold the recheck until this is merged - https://review.opendev.org/#/c/734700/ -gmann
Testing Status: =========== * ~270 repos gate have been tested green or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~28 repos are failing. Need immediate action mentioned above. ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~18repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
-gmann
-gmann
-gmann
---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann@ghanshyammann.com> wrote ----
Hello Everyone,
Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can break the projects gate if not yet taken care of. Read below for the plan.
Tracking: https://storyboard.openstack.org/#!/story/2007865
Progress: ======= * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below plan.
* Part1: Migrating tox base job tomorrow (8th Sept):
** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this series (all base patches of this): https://review.opendev.org/#/c/738328/ .
**There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible. Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing.
* Part2: Migrating devstack/tempest base job on 10th sept:
* We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept.
** Bug#1882521 ** DB migration issues, *** alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
Testing Till now: ============ * ~200 repos gate have been tested or fixed till now. ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+s...)
* ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your project repos if I am late to fix them): ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
* ~30repos fixes ready to merge: ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVe...
Bugs Report: ==========
1. Bug#1882521. (IN-PROGRESS) There is open bug for nova/cinder where three tempest tests are failing for volume detach operation. There is no clear root cause found yet -https://bugs.launchpad.net/cinder/+bug/1882521 We have skipped the tests in tempest base patch to proceed with the other projects testing but this is blocking things for the migration.
2. DB migration issues (IN-PROGRESS) * alembic and few on telemetry/gnocchi side https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
3. We encountered the nodeset name conflict with x/tobiko. (FIXED) nodeset conflict is resolved now and devstack provides all focal nodes now.
4. Bug#1886296. (IN-PROGRESS) pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8. As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1 on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3] nd will release a new hacking version. After that project can move to new hacking and do not need to maintain pyflakes version compatibility.
5. Bug#1886298. (IN-PROGRESS) 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4], We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work. There are a few more issues[5] with lower-constraint jobs which I am debugging.
What work to be done on the project side: ================================ This goal is more of testing the jobs on focal and fixing bugs if any otherwise migrate jobs by switching the nodeset to focal node sets defined in devstack.
1. Start a patch in your repo by making depends-on on either of below: devstack base patch if you are using only devstack base jobs not tempest:
Depends-on: https://review.opendev.org/#/c/731207/ OR tempest base patch if you are using the tempest base job (like devstack-tempest): Depends-on: https://review.opendev.org/#/c/734700/
Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So you can test the complete gate jobs(unit/functional/doc/integration) together. This and its base patches - https://review.opendev.org/#/c/738328/
Example: https://review.opendev.org/#/c/738126/
2. If none of your project jobs override the nodeset then above patch will be testing patch(do not merge) otherwise change the nodeset to focal. Example: https://review.opendev.org/#/c/737370/
3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset is overridden then devstack being branched and stable base job using bionic/xenial will take care of this. Example: https://review.opendev.org/#/c/744056/2
4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in this migration. Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
Once we finish the testing on projects side and no failure then we will merge the devstack and tempest base patches.
Important things to note: =================== * Do not forgot to add the story and task link to your patch so that we can track it smoothly. * Use gerrit topic 'migrate-to-focal' * Do not backport any of the patches.
References: ========= Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jo... Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
[1] https://github.com/PyCQA/pyflakes/issues/367 [2] https://review.opendev.org/#/c/739315/ [3] https://review.opendev.org/#/c/739334/ [4] https://github.com/pallets/markupsafe/issues/116 [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
-gmann
participants (7)
-
Ghanshyam Mann
-
Jeremy Stanley
-
Lee Yarwood
-
Matthias Runge
-
Michał Dulko
-
Radosław Piliszek
-
Yasufumi Ogawa