Open Stack

 ---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 > Updates:
 > After working more on failing one today and listing the blocking one, I think we are good to switch tox based testing today
 > and discuss the integration testing switch tomorrow in TC office hours. 
 > 
 > > * Part1: Migrating tox base job tomorrow (8th Sept):

This is done and almost all the projects are fixed or at least fixes are up to merge. cinder and keystone l-c job on Focal also working fine and ready to merge.

Few python clients have not yet merged the fixes so I have backported those to Victoria to fix master as well as Victoria gate.

[...]
 >  > * Part2: Migrating devstack/tempest  base job on 10th sept:
 > We have three blocking open bugs here so I would like to discuss it in tomorrow's TC office hour also about how to proceed on this.

 > 1. Nova: https://bugs.launchpad.net/nova/+bug/1882521  (https://bugs.launchpad.net/qemu/+bug/1894804)
As of now, no work needed on Nova side and after QEMU fix, we will see if tests pass.

 > 2. Barbican: https://storyboard.openstack.org/#!/story/2007732
The only blocker left for Focal migration. 

 > 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
This worked fine with mariadb - https://review.opendev.org/#/c/752294/

There are many fixes still not merged yet and their gate is also failing, I request to merge the fixes on priority:
- https://review.opendev.org/#/q/topic:migrate-to-focal+status:open

-gmann

 > 
 > 
 > -gmann
 > 
 > 
 >  ---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
 >  > Hello Everyone,
 >  > 
 >  > Please find the week R-4 updates on 'Ubuntu Focal migration' community goal. Its time to force the base jobs migration which can
 >  > break the projects gate if not yet taken care of. Read below for the plan.
 >  > 
 >  > Tracking: https://storyboard.openstack.org/#!/story/2007865
 >  > 
 >  > Progress:
 >  > =======
 >  > * We are close to V-3 release and this is time we have to complete this migration otherwise doing it in RC period can add
 >  > unnecessary and last min delay. I am going to plan this migration in two-part. This will surely break some projects gate
 >  > which is not yet finished the migration but we have to do at some time. Please let me know if any objection to the below
 >  > plan.
 >  > 
 >  > * Part1: Migrating tox base job tomorrow (8th Sept):
 >  > 
 >  > ** I am going to open tox base jobs migration (doc, unit, functional, lower-constraints etc) to merge by tomorrow. which is this
 >  > series (all base patches of this):  https://review.opendev.org/#/c/738328/ . 
 >  > 
 >  > **There are few repos still failing on requirements lower-constraints job specifically which I tried my best to fix as many as possible.
 >  >  Many are ready to merge also. Please merge or work on your projects repo testing before that or fix on priority if failing.
 >  > 
 >  > * Part2: Migrating devstack/tempest  base job on 10th sept:
 >  > 
 >  > * We have few open bugs for this which are not yet resolved, we will see how it goes but the current plan is to migrate by 10th Sept.
 >  > 
 >  > ** Bug#1882521
 >  > ** DB migration issues,
 >  > *** alembic and few on telemetry/gnocchi side  https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
 >  > 
 >  > 
 >  > Testing Till now:
 >  > ============
 >  > * ~200 repos gate have been tested or fixed till now.
 >  > ** https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+status:merged)
 >  > 
 >  > * ~100 repos are under test and failing. Debugging and fixing are in progress (If you would like to help, please check your
 >  > project repos if I am late to fix them):
 >  > ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
 >  > 
 >  > * ~30repos fixes ready to merge:
 >  > ** https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVerified%3E%3D1%2Czuul+NOT+label%3AWorkflow%3C%3D-1
 >  > 
 >  > 
 >  > Bugs Report:
 >  > ==========
 >  > 
 >  > 1. Bug#1882521. (IN-PROGRESS)
 >  > There is open bug for nova/cinder where three tempest tests are failing for
 >  > volume detach operation. There is no clear root cause found yet
 >  > -https://bugs.launchpad.net/cinder/+bug/1882521
 >  > We have skipped the tests in tempest base patch to proceed with the other
 >  > projects testing but this is blocking things for the migration.
 >  > 
 >  > 2. DB migration issues (IN-PROGRESS)
 >  > * alembic and few on telemetry/gnocchi side  https://github.com/sqlalchemy/alembic/issues/699, https://storyboard.openstack.org/#!/story/2008003
 >  > 
 >  > 3. We encountered the nodeset name conflict with x/tobiko. (FIXED)
 >  > nodeset conflict is resolved now and devstack provides all focal nodes now.
 >  > 
 >  > 4. Bug#1886296. (IN-PROGRESS)
 >  > pyflakes till 2.1.0 is not compatible with python 3.8 which is the default python version
 >  > on ubuntu focal[1]. With pep8 job running on focal faces the issue and fail. We need to bump
 >  > the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8.
 >  > As of now, many projects are using old hacking version so I am explicitly adding pyflakes>=2.1.1
 >  > on the project side[2] but for the long term easy maintenance, I am doing it in 'hacking' requirements.txt[3]
 >  > nd will release a new hacking version. After that project can move to new hacking and do not need
 >  > to maintain pyflakes version compatibility.
 >  > 
 >  > 5. Bug#1886298. (IN-PROGRESS)
 >  > 'Markupsafe' 1.0 is not compatible with the latest version of setuptools[4],
 >  > We need to bump the lower-constraint for Markupsafe to 1.1.1 to make it work.
 >  > There are a few more issues[5] with lower-constraint jobs which I am debugging.
 >  > 
 >  > 
 >  > What work to be done on the project side:
 >  > ================================
 >  > This goal is more of testing the jobs on focal and fixing bugs if any otherwise
 >  > migrate jobs by switching the nodeset to focal node sets defined in devstack.
 >  > 
 >  > 1. Start a patch in your repo by making depends-on on either of below:
 >  > devstack base patch if you are using only devstack base jobs not tempest:
 >  > 
 >  > Depends-on: https://review.opendev.org/#/c/731207/
 >  > OR
 >  > tempest base patch if you are using the tempest base job (like devstack-tempest):
 >  > Depends-on: https://review.opendev.org/#/c/734700/
 >  > 
 >  > Both have depends-on on the series where I am moving unit/functional/doc/cover/nodejs tox jobs to focal. So
 >  > you can test the complete gate jobs(unit/functional/doc/integration) together.
 >  > This and its base patches - https://review.opendev.org/#/c/738328/
 >  > 
 >  > Example: https://review.opendev.org/#/c/738126/
 >  > 
 >  > 2. If none of your project jobs override the nodeset then above patch will be
 >  > testing patch(do not merge) otherwise change the nodeset to focal.
 >  > Example: https://review.opendev.org/#/c/737370/
 >  > 
 >  > 3. If the jobs are defined in branchless repo and override the nodeset then you need to override the branches
 >  > variant to adjust the nodeset so that those jobs run on Focal on victoria onwards only. If no nodeset
 >  > is overridden then devstack being branched and stable base job using bionic/xenial will take care of
 >  > this.
 >  > Example: https://review.opendev.org/#/c/744056/2
 >  > 
 >  > 4. If no updates need you can abandon the testing patch (https://review.opendev.org/#/c/744341/). If it need
 >  > updates then modify the same patch with proper commit msg, once it pass the gate then remove the Depends-On
 >  > so that you can merge your patch before base jobs are switched to focal. This way we make sure no gate downtime in
 >  > this migration.
 >  > Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
 >  > 
 >  > Once we finish the testing on projects side and no failure then we will merge the devstack and tempest
 >  > base patches.
 >  > 
 >  > 
 >  > Important things to note:
 >  > ===================
 >  > * Do not forgot to add the story and task link to your patch so that we can track it smoothly.
 >  > * Use gerrit topic 'migrate-to-focal'
 >  > * Do not backport any of the patches.
 >  > 
 >  > 
 >  > References:
 >  > =========
 >  > Goal doc: https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jobs-to-ubuntu-focal.html
 >  > Storyboard tracking: https://storyboard.openstack.org/#!/story/2007865
 >  > 
 >  > [1] https://github.com/PyCQA/pyflakes/issues/367
 >  > [2] https://review.opendev.org/#/c/739315/
 >  > [3] https://review.opendev.org/#/c/739334/
 >  > [4] https://github.com/pallets/markupsafe/issues/116
 >  > [5] https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
 >  > 
 >  > -gmann
 >  > 
 >  > 
 >