Open Stack

On Wed, 2020-09-09 at 14:05 -0500, Ghanshyam Mann wrote:
>  ---- On Tue, 08 Sep 2020 17:56:05 -0500 Ghanshyam Mann <
> gmann at ghanshyammann.com> wrote ----
>  > Updates:
>  > After working more on failing one today and listing the blocking
> one, I think we are good to switch tox based testing today
>  > and discuss the integration testing switch tomorrow in TC office
> hours. 
>  > 
>  > > * Part1: Migrating tox base job tomorrow (8th Sept):
>  >  
>  > I have checked it again and fixed many repos that are up for
> review and merge. Most python clients are already fixed
>  > or their fixes are up for merge so they can make it before the
> feature freeze on 10th. If any repo is broken then it will be pretty
> quick
>  > to fix by lower constraint bump (see the example under 
> https://review.opendev.org/#/q/topic:migrate-to-focal)
>  > 
>  > Even if any of the fixes miss the victoria release then those can
> be backported easily. I am opening the tox base jobs migration to
> merge: 
>  >  - All patches in this series 
> https://review.opendev.org/#/c/738328/
> 
> All these tox base jobs are merged now and running on Focal. If any
> of your repo is failing, please fix on priority or ping me on IRC if
> failure not clear.
> You can find most of the fixes for possible failure in this topic:
> - 
> https://review.opendev.org/#/q/topic:migrate-to-focal+(status:open+OR+status:merged)
> 
> -gmann

We're in a bit of a pickle here. So with kuryr-kubernetes we aim to
keep lower-constraints on the versions that can be found in
CentOS/RHEL8 and seems like cffi 1.11.5 won't compile with Python 3.8.
What should we do here? Is such assumption even possible given broader
OpenStack assumptions?

>  > 
>  > 
>  >  > * Part2: Migrating devstack/tempest  base job on 10th sept:
>  > 
>  > We have three blocking open bugs here so I would like to discuss
> it in tomorrow's TC office hour also about how to proceed on this.
>  > 
>  > 1. Nova: https://bugs.launchpad.net/nova/+bug/1882521  (
> https://bugs.launchpad.net/qemu/+bug/1894804)
>  > 2. Barbican: https://storyboard.openstack.org/#!/story/2007732
>  > 3. Ceilometer: https://storyboard.openstack.org/#!/story/2008121
>  > 
>  > 
>  > -gmann
>  > 
>  > 
>  >  ---- On Mon, 07 Sep 2020 09:29:40 -0500 Ghanshyam Mann <
> gmann at ghanshyammann.com> wrote ----
>  >  > Hello Everyone,
>  >  > 
>  >  > Please find the week R-4 updates on 'Ubuntu Focal migration'
> community goal. Its time to force the base jobs migration which can
>  >  > break the projects gate if not yet taken care of. Read below
> for the plan.
>  >  > 
>  >  > Tracking: https://storyboard.openstack.org/#!/story/2007865
>  >  > 
>  >  > Progress:
>  >  > =======
>  >  > * We are close to V-3 release and this is time we have to
> complete this migration otherwise doing it in RC period can add
>  >  > unnecessary and last min delay. I am going to plan this
> migration in two-part. This will surely break some projects gate
>  >  > which is not yet finished the migration but we have to do at
> some time. Please let me know if any objection to the below
>  >  > plan.
>  >  > 
>  >  > * Part1: Migrating tox base job tomorrow (8th Sept):
>  >  > 
>  >  > ** I am going to open tox base jobs migration (doc, unit,
> functional, lower-constraints etc) to merge by tomorrow. which is
> this
>  >  > series (all base patches of this):  
> https://review.opendev.org/#/c/738328/ . 
>  >  > 
>  >  > **There are few repos still failing on requirements lower-
> constraints job specifically which I tried my best to fix as many as
> possible.
>  >  >  Many are ready to merge also. Please merge or work on your
> projects repo testing before that or fix on priority if failing.
>  >  > 
>  >  > * Part2: Migrating devstack/tempest  base job on 10th sept:
>  >  > 
>  >  > * We have few open bugs for this which are not yet resolved, we
> will see how it goes but the current plan is to migrate by 10th Sept.
>  >  > 
>  >  > ** Bug#1882521
>  >  > ** DB migration issues,
>  >  > *** alembic and few on telemetry/gnocchi side  
> https://github.com/sqlalchemy/alembic/issues/699, 
> https://storyboard.openstack.org/#!/story/2008003
>  >  > 
>  >  > 
>  >  > Testing Till now:
>  >  > ============
>  >  > * ~200 repos gate have been tested or fixed till now.
>  >  > ** 
> https://review.opendev.org/#/q/topic:migrate-to-focal+(status:abandoned+OR+status:merged)
>  >  > 
>  >  > * ~100 repos are under test and failing. Debugging and fixing
> are in progress (If you would like to help, please check your
>  >  > project repos if I am late to fix them):
>  >  > ** 
> https://review.opendev.org/#/q/topic:migrate-to-focal+status:open
>  >  > 
>  >  > * ~30repos fixes ready to merge:
>  >  > ** 
> https://review.opendev.org/#/q/topic:migrate-to-focal+status:open+label%3AVerified%3E%3D1%2Czuul+NOT+label%3AWorkflow%3C%3D-1
>  >  > 
>  >  > 
>  >  > Bugs Report:
>  >  > ==========
>  >  > 
>  >  > 1. Bug#1882521. (IN-PROGRESS)
>  >  > There is open bug for nova/cinder where three tempest tests are
> failing for
>  >  > volume detach operation. There is no clear root cause found yet
>  >  > -https://bugs.launchpad.net/cinder/+bug/1882521
>  >  > We have skipped the tests in tempest base patch to proceed with
> the other
>  >  > projects testing but this is blocking things for the migration.
>  >  > 
>  >  > 2. DB migration issues (IN-PROGRESS)
>  >  > * alembic and few on telemetry/gnocchi side  
> https://github.com/sqlalchemy/alembic/issues/699, 
> https://storyboard.openstack.org/#!/story/2008003
>  >  > 
>  >  > 3. We encountered the nodeset name conflict with x/tobiko.
> (FIXED)
>  >  > nodeset conflict is resolved now and devstack provides all
> focal nodes now.
>  >  > 
>  >  > 4. Bug#1886296. (IN-PROGRESS)
>  >  > pyflakes till 2.1.0 is not compatible with python 3.8 which is
> the default python version
>  >  > on ubuntu focal[1]. With pep8 job running on focal faces the
> issue and fail. We need to bump
>  >  > the pyflakes to 2.1.1 as min version to run pep8 jobs on py3.8.
>  >  > As of now, many projects are using old hacking version so I am
> explicitly adding pyflakes>=2.1.1
>  >  > on the project side[2] but for the long term easy maintenance,
> I am doing it in 'hacking' requirements.txt[3]
>  >  > nd will release a new hacking version. After that project can
> move to new hacking and do not need
>  >  > to maintain pyflakes version compatibility.
>  >  > 
>  >  > 5. Bug#1886298. (IN-PROGRESS)
>  >  > 'Markupsafe' 1.0 is not compatible with the latest version of
> setuptools[4],
>  >  > We need to bump the lower-constraint for Markupsafe to 1.1.1 to
> make it work.
>  >  > There are a few more issues[5] with lower-constraint jobs which
> I am debugging.
>  >  > 
>  >  > 
>  >  > What work to be done on the project side:
>  >  > ================================
>  >  > This goal is more of testing the jobs on focal and fixing bugs
> if any otherwise
>  >  > migrate jobs by switching the nodeset to focal node sets
> defined in devstack.
>  >  > 
>  >  > 1. Start a patch in your repo by making depends-on on either of
> below:
>  >  > devstack base patch if you are using only devstack base jobs
> not tempest:
>  >  > 
>  >  > Depends-on: https://review.opendev.org/#/c/731207/
>  >  > OR
>  >  > tempest base patch if you are using the tempest base job (like
> devstack-tempest):
>  >  > Depends-on: https://review.opendev.org/#/c/734700/
>  >  > 
>  >  > Both have depends-on on the series where I am moving
> unit/functional/doc/cover/nodejs tox jobs to focal. So
>  >  > you can test the complete gate
> jobs(unit/functional/doc/integration) together.
>  >  > This and its base patches - 
> https://review.opendev.org/#/c/738328/
>  >  > 
>  >  > Example: https://review.opendev.org/#/c/738126/
>  >  > 
>  >  > 2. If none of your project jobs override the nodeset then above
> patch will be
>  >  > testing patch(do not merge) otherwise change the nodeset to
> focal.
>  >  > Example: https://review.opendev.org/#/c/737370/
>  >  > 
>  >  > 3. If the jobs are defined in branchless repo and override the
> nodeset then you need to override the branches
>  >  > variant to adjust the nodeset so that those jobs run on Focal
> on victoria onwards only. If no nodeset
>  >  > is overridden then devstack being branched and stable base job
> using bionic/xenial will take care of
>  >  > this.
>  >  > Example: https://review.opendev.org/#/c/744056/2
>  >  > 
>  >  > 4. If no updates need you can abandon the testing patch (
> https://review.opendev.org/#/c/744341/). If it need
>  >  > updates then modify the same patch with proper commit msg, once
> it pass the gate then remove the Depends-On
>  >  > so that you can merge your patch before base jobs are switched
> to focal. This way we make sure no gate downtime in
>  >  > this migration.
>  >  > Example: https://review.opendev.org/#/c/744056/1..2//COMMIT_MSG
>  >  > 
>  >  > Once we finish the testing on projects side and no failure then
> we will merge the devstack and tempest
>  >  > base patches.
>  >  > 
>  >  > 
>  >  > Important things to note:
>  >  > ===================
>  >  > * Do not forgot to add the story and task link to your patch so
> that we can track it smoothly.
>  >  > * Use gerrit topic 'migrate-to-focal'
>  >  > * Do not backport any of the patches.
>  >  > 
>  >  > 
>  >  > References:
>  >  > =========
>  >  > Goal doc: 
> https://governance.openstack.org/tc/goals/selected/victoria/migrate-ci-cd-jobs-to-ubuntu-focal.html
>  >  > Storyboard tracking: 
> https://storyboard.openstack.org/#!/story/2007865
>  >  > 
>  >  > [1] https://github.com/PyCQA/pyflakes/issues/367
>  >  > [2] https://review.opendev.org/#/c/739315/
>  >  > [3] https://review.opendev.org/#/c/739334/
>  >  > [4] https://github.com/pallets/markupsafe/issues/116
>  >  > [5] 
> https://zuul.opendev.org/t/openstack/build/7ecd9cf100194bc99b3b70fa1e6de032
>  >  > 
>  >  > -gmann
>  >  > 
>  >  > 
>  > 
>  > 
>