[all][tc][goals] Migrate CI/CD jobs to new Ubuntu LTS Focal: Week R-4 Update

Ghanshyam Mann gmann at ghanshyammann.com
Fri Sep 11 12:30:28 UTC 2020


 ---- On Fri, 11 Sep 2020 07:13:14 -0500 Radosław Piliszek <radoslaw.piliszek at gmail.com> wrote ----
 > I agree with Michał that it kind of breaks the purpose of lower-constraints.
 > Supposedly lower-constraints should just be tested with the lowest
 > supported python version?
 > WDYT, Folks?
 > 
 > (That said, lots of projects already made lower-constraints break on
 > RDO due to these bumps.)

This is something we discussed yesterday, there are both way we can argue whether
we should test l-c on lower supported python or available python on tested Disro which
we do with Ubuntu Focal for all tox based jobs. And believe be lower constraints are not
only python version things. But It is fine for projects say kuryr-kubernetes to keep running
the l-c job on Bionic or centos I can provide patch for that.

and for RHEL compatible version, it can be adjusted as I did in ceilometer for lxml
but I am not sure if we can test all of them for RHEL also.
- https://review.opendev.org/#/c/744612/

I am going to submit a Forum session to discuss on this topic so that we have
an agreed way of testing in future,  

-gman


 > 
 > -yoctozepto
 > 
 > On Fri, Sep 11, 2020 at 2:02 PM Michał Dulko <mdulko at redhat.com> wrote:
 > >
 > > On Wed, 2020-09-09 at 14:05 -0500, Ghanshyam Mann wrote:
 > > >  ---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann <
 > > > gmann at ghanshyammann.com> wrote ----
 > > >  > Updates:
 > > >  > After working more on failing one today and listing the blocking
 > > > one, I think we are good to switch tox based testing today
 > > >  > and discuss the integration testing switch tomorrow in TC office
 > > > hours.
 > > >  >
 > > >  > > * Part1: Migrating tox base job tomorrow (8th Sept):
 > > >  >
 > > >  > I have checked it again and fixed many repos that are up for
 > > > review and merge. Most python clients are already fixed
 > > >  > or their fixes are up for merge so they can make it before the
 > > > feature freeze on 10th. If any repo is broken then it will be pretty
 > > > quick
 > > >  > to fix by lower constraint bump (see the example under
 > > > https://review.opendev.org/#/q/topic:migrate-to-focal)
 > > >  >
 > > >  > Even if any of the fixes miss the victoria release then those can
 > > > be backported easily. I am opening the tox base jobs migration to
 > > > merge:
 > > >  >  - All patches in this series
 > > > https://review.opendev.org/#/c/738328/
 > > >
 > > > All these tox base jobs are merged now and running on Focal. If any
 > > > of your repo is failing, please fix on priority or ping me on IRC if
 > > > failure not clear.
 > > > You can find most of the fixes for possible failure in this topic:
 > > > -
 > > > https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status:merged)
 > > >
 > > > -gmann
 > >
 > > We're in a bit of a pickle here. So with kuryr-kubernetes we aim to
 > > keep lower-constraints on the versions that can be found in
 > > CentOS/RHEL8 and seems like cffi 1.11.5 won't compile with Python 3.8.
 > > What should we do here? Is such assumption even possible given broader
 > > OpenStack assumptions?
 > >
 > > >  >
 > > >  >
 > > >  >  > * Part2: Migrating devstack/tempest  base job on 10th sept:
 > > >  >
 > > >  > We have three blocking open bugs here so I would like to discuss
 > > > it in tomorrow's TC office hour also about how to proceed on this.
 > > >  >
 > > >  > 1. Nova: https://bugs.launchpad.net/nova/+bug/1882521  (
 > > > https://bugs.launchpad.net/qemu/+bug/1894804)
 > > >  > 2. Barbican: https://storyboard.openstack.org/#!/story/2007732
 > > >  > 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
 > > >  >
 > > >  >
 > > >  > -gmann
 > > >  >
 > > >  >
 > > >  >  ---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <
 > > > gmann at ghanshyammann.com> wrote ----
 > > >  >  > Hello Everyone,
 > > >  >  >
 > > >  >  > Please find the week R-4 updates on 'Ubuntu Focal migration'
 > > > community goal. Its time to force the base jobs migration which can
 > > >  >  > break the projects gate if not yet taken care of. Read below
 > > > for the plan.
 > > >  >  >
 > > >  >  > Tracking: https://storyboard.openstack.org/#!/story/2007865
 > > >  >  >
 > > >  >  > Progress:
 > > >  >  > =======
 > > >  >  > * We are close to V-3 release and this is time we have to
 > > > complete this migration otherwise doing it in RC period can add
 > > >  >  > unnecessary and last min delay. I am going to plan this
 > > > migration in two-part. This will surely break some projects gate
 > > >  >  > which is not yet finished the migration but we have to do at
 > > > some time. Please let me know if any objection to the below
 > > >  >  > plan.
 > > >  >  >
 > > >  >  > * Part1: Migrating tox base job tomorrow (8th Sept):
 > > >  >  >
 > > >  >  > ** I am going to open tox base jobs migration (doc, unit,
 > > > functional, lower-constraints etc) to merge by tomorrow. which is
 > > > this
 > > >  >  > series (all base patches of this):
 > > > https://review.opendev.org/#/c/738328/ .
 > > >  >  >
 > > >  >  > **There are few repos still failing on requirements lower-
 > > > constraints job specifically which I tried my best to fix as many as
 > > > possible.
 > > >  >  >  Many are ready to merge also. Please merge or work on your
 > > > projects repo testing before that or fix on priority if failing.
 > > >  >  >
 > > >  >  > * Part2: Migrating devstack/tempest  base job on 10th sept:
 > > >  >  >
 > > >  >  > * We have few open bugs for this which are not yet resolved, we
 > > > will see how it goes but the current plan is to migrate by 10th Sept.
 > > >  >  >
 > > >  >  > ** Bug#1882521
 > > >  >  > ** DB migration issues,
 > > >  >  > *** alembic and few on telemetry/gnocchi side
 > > > https://github.com/sqlalchemy/alembic/issues/699,
 > > > https://storyboard.openstack.org/#!/story/2008003
 > > >  >  >
 > > >  >  >
 > > >  >  > Testing Till now:
 > > >  >  > ============
 > > >  >  > * ~200 repos gate have been tested or fixed till now.
 > > >  >  > **
 > > > https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+status:merged)
 > > >  >  >
 > > >  >  > * ~100 repos are under test and failing. Debugging and fixing
 > > > are in progress (If you would like to help, please check your
 > > >  >  > project repos if I am late to fix them):
 > > >  >  > **
 > > > https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
 > > >  >  >
 > > >  >  > * ~30repos fixes ready to merge:
 > > >  >  > **
 > > > https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVerified%3E%3D1%2Czuul+NOT+label%3AWorkflow%3C%3D-1
 > > >  >  >
 > > >  >  >
 > > >  >  > Bugs Report:
 > > >  >  > ==========
 > > >  >  >
 > > >  >  > 1. Bug#1882521. (IN-PROGRESS)
 > > >  >  > There is open bug for nova/cinder where three tempest tests are
 > > > failing for
 > > >  >  > volume detach operation. There is no clear root cause found yet
 > > >  >  > -https://bugs.launchpad.net/cinder/+bug/1882521
 > > >  >  > We have skipped the tests in tempest base patch to proceed with
 > > > the other
 > > >  >  > projects testing but this is blocking things for the migration.
 > > >  >  >
 > > >  >  > 2. DB migration issues (IN-PROGRESS)
 > > >  >  > * alembic and few on telemetry/gnocchi side
 > > > https://github.com/sqlalchemy/alembic/issues/699,
 > > > https://storyboard.openstack.org/#!/story/2008003
 > > >  >  >
 > > >  >  > 3. We encountered the nodeset name conflict with x/tobiko.
 > > > (FIXED)
 > > >  >  > nodeset conflict is resolved now and devstack provides all
 > > > focal nodes now.
 > > >  >  >
 > > >  >  > 4. Bug#1886296. (IN-PROGRESS)
 > > >  >  > pyflakes till 2.1.0 is not compatible with python 3.8 which is
 > > > the default python version
 > > >  >  > on ubuntu focal[1]. With pep8 job running on focal faces the
 > > > issue and fail. We need to bump
 > > >  >  > the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8.
 > > >  >  > As of now, many projects are using old hacking version so I am
 > > > explicitly adding pyflakes>=2.1.1
 > > >  >  > on the project side[2] but for the long term easy maintenance,
 > > > I am doing it in 'hacking' requirements.txt[3]
 > > >  >  > nd will release a new hacking version. After that project can
 > > > move to new hacking and do not need
 > > >  >  > to maintain pyflakes version compatibility.
 > > >  >  >
 > > >  >  > 5. Bug#1886298. (IN-PROGRESS)
 > > >  >  > 'Markupsafe' 1.0 is not compatible with the latest version of
 > > > setuptools[4],
 > > >  >  > We need to bump the lower-constraint for Markupsafe to 1.1.1 to
 > > > make it work.
 > > >  >  > There are a few more issues[5] with lower-constraint jobs which
 > > > I am debugging.
 > > >  >  >
 > > >  >  >
 > > >  >  > What work to be done on the project side:
 > > >  >  > ================================
 > > >  >  > This goal is more of testing the jobs on focal and fixing bugs
 > > > if any otherwise
 > > >  >  > migrate jobs by switching the nodeset to focal node sets
 > > > defined in devstack.
 > > >  >  >
 > > >  >  > 1. Start a patch in your repo by making depends-on on either of
 > > > below:
 > > >  >  > devstack base patch if you are using only devstack base jobs
 > > > not tempest:
 > > >  >  >
 > > >  >  > Depends-on: https://review.opendev.org/#/c/731207/
 > > >  >  > OR
 > > >  >  > tempest base patch if you are using the tempest base job (like
 > > > devstack-tempest):
 > > >  >  > Depends-on: https://review.opendev.org/#/c/734700/
 > > >  >  >
 > > >  >  > Both have depends-on on the series where I am moving
 > > > unit/functional/doc/cover/nodejs tox jobs to focal. So
 > > >  >  > you can test the complete gate
 > > > jobs(unit/functional/doc/integration) together.
 > > >  >  > This and its base patches -
 > > > https://review.opendev.org/#/c/738328/
 > > >  >  >
 > > >  >  > Example: https://review.opendev.org/#/c/738126/
 > > >  >  >
 > > >  >  > 2. If none of your project jobs override the nodeset then above
 > > > patch will be
 > > >  >  > testing patch(do not merge) otherwise change the nodeset to
 > > > focal.
 > > >  >  > Example: https://review.opendev.org/#/c/737370/
 > > >  >  >
 > > >  >  > 3. If the jobs are defined in branchless repo and override the
 > > > nodeset then you need to override the branches
 > > >  >  > variant to adjust the nodeset so that those jobs run on Focal
 > > > on victoria onwards only. If no nodeset
 > > >  >  > is overridden then devstack being branched and stable base job
 > > > using bionic/xenial will take care of
 > > >  >  > this.
 > > >  >  > Example: https://review.opendev.org/#/c/744056/2
 > > >  >  >
 > > >  >  > 4. If no updates need you can abandon the testing patch (
 > > > https://review.opendev.org/#/c/744341/). If it need
 > > >  >  > updates then modify the same patch with proper commit msg, once
 > > > it pass the gate then remove the Depends-On
 > > >  >  > so that you can merge your patch before base jobs are switched
 > > > to focal. This way we make sure no gate downtime in
 > > >  >  > this migration.
 > > >  >  > Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
 > > >  >  >
 > > >  >  > Once we finish the testing on projects side and no failure then
 > > > we will merge the devstack and tempest
 > > >  >  > base patches.
 > > >  >  >
 > > >  >  >
 > > >  >  > Important things to note:
 > > >  >  > ===================
 > > >  >  > * Do not forgot to add the story and task link to your patch so
 > > > that we can track it smoothly.
 > > >  >  > * Use gerrit topic 'migrate-to-focal'
 > > >  >  > * Do not backport any of the patches.
 > > >  >  >
 > > >  >  >
 > > >  >  > References:
 > > >  >  > =========
 > > >  >  > Goal doc:
 > > > https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jobs-to-ubuntu-focal.html
 > > >  >  > Storyboard tracking:
 > > > https://storyboard.openstack.org/#!/story/2007865
 > > >  >  >
 > > >  >  > [1] https://github.com/PyCQA/pyflakes/issues/367
 > > >  >  > [2] https://review.opendev.org/#/c/739315/
 > > >  >  > [3] https://review.opendev.org/#/c/739334/
 > > >  >  > [4] https://github.com/pallets/markupsafe/issues/116
 > > >  >  > [5]
 > > > https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
 > > >  >  >
 > > >  >  > -gmann
 > > >  >  >
 > > >  >  >
 > > >  >
 > > >  >
 > > >
 > >
 > >
 > >
 > 



More information about the openstack-discuss mailing list