[all][tc] Planning for dropping the Python2 support in OpenStack
Ghanshyam Mann
gmann at ghanshyammann.com
Wed Nov 20 08:08:52 UTC 2019
---- On Tue, 19 Nov 2019 19:30:52 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
> ---- On Tue, 19 Nov 2019 10:31:51 -0600 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
> > Hello Everyone,
> >
> > I would like to notify all projects about important discussions and agreement happened today to
> > move forward for cross projects dependencies and devstack default installation or py2 drop work.
> >
> > * It is now an official community goal for ussuri[1] and except Swift, all projects agreed to drop py2
> > as per schedule in goal.
> >
> > * If any project (openstack services which are planned to drop from now till m-1) drop the py2 with removing
> > the py2 requirements and min python version in setup.cfg which makes that project uninstallable in cross
> > projects jobs then:
> >
> > Options 1 (suggested): broken projects has to drop the py2 support/testing immediately.
> >
> > Options 2: if it breaks most of the projects, for example, nova or any other default projects become uninstallable
> > on py2 then we can half-revert[2] the changes from the project caused the break and wait till m1 to
> > merge them back.
> >
> > * Making Devstack to py3 by default TODAY (otherwise it can break gate everyday).
> > **Devstack default is py2 currently and it was planned to make py3 by default after m-1. But after seeing today gate break, it is hard
> > to maintain devstack-py2-by-default. because projects are dropping the py2 support and devstack py2 by default cause the
> > problem[3]. Today it is from nova side and It can happen due to any projects dropping py2 or I should say it can happen every day as
> > py2 drop patches get merged.
> > ** I am ok to make Devstack py3 by default today which is this patch - https://review.opendev.org/#/c/649097/
> > ** Action for projects who want to keep testing py2 job till m-1 or whenever they plan to drop py2: Explicitly disable
> > the py3 in their py2 jobs (USE_PYTHON3: False).
>
> Devstack patch moving to py3 by default is approved and waiting for base patches to merge first. I am monitoring that
> to be merged by night.
>
> One thing it will break for sure is grenade jobs which we have seen in the same patch also. After auditing all the projects
> for grenade py2 jobs, patches to move those jobs to py3 is up[1]. grenade jobs in openstack-zuul-jobs are mainly for stable branches
> so I kept them on py2 by explicitly disable the py3. later while dropping the py2, I will move those jobs to projects side with py3 version.
>
> If you find your project gate broken for grenade job then, you need to merge the patches which are already up[1].
>
> If any other job is broken then,
>
> Option1: migrate the failing py2 jobs to py3 with 'USE_PYTHON3: True in zuulv3 jobs' and 'DEVSTACK_GATE_USE_PYTHON3=True in legacy jobs'.
>
> Options2: If the migration to py3 takes time due to any reason then restore those jobs to py2 by explicitly disabled the py3 via above variables.
>
>
> [1] https://review.opendev.org/#/q/topic:drop-py27-support-devstack-default-py3+(status:open+OR+status:merged)
Devstack patch is about to merge, Below is the status of grenade jobs fixes:
* heat- https://review.opendev.org/#/c/695088/
- One test is failing on grenade job consistently. Not sure how that is related to py3.
* barbican- https://review.opendev.org/#/c/695052/
- Some strange behaviour is happening in this, barbican grenade job run stable/train run.yaml always which is stoping this job to move to py3
* Octavia- https://review.opendev.org/#/c/693486/3
- This needs to be updated to mention the py3 version in run.yaml.
* rest all other patches are passing gate so merge them asap before devstack patch is merged and block their gate.
-gmann
>
> -gmann
>
> >
> > * I have pushed the py2 drop patches on almost all the OpenStack services[4] which migrate py2 jobs to py3, remove the py2
> > requirement but do not mention t
> he min python version in setup.cfg (which can ben done by followup if projects want to
> > do that). I will suggest to merge them asap to avoid any gate break due to cross projects dependency.
> >
> >
> > [1] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html
> > [2] half-revert means only change in requirements.txt and setup.cfg - https://review.opendev.org/#/c/695007/
> > [3] https://bugs.launchpad.net/nova/+bug/1853166
> > [4] https://review.opendev.org/#/q/topic:drop-py27-support+(status:open+OR+status:merged)
> >
> > -gmann
> >
> >
> > ---- On Wed, 30 Oct 2019 12:03:33 -0500 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
> > > ---- On Wed, 30 Oct 2019 06:59:19 -0500 Sean Mooney <smooney at redhat.com> wrote ----
> > > > On Tue, 2019-10-29 at 19:40 -0500, Matthew Thode wrote:
> > > > > On 19-10-29 14:53:11, Ghanshyam Mann wrote:
> > > > > > ---- On Thu, 24 Oct 2019 14:32:03 -0500 Ghanshyam Mann <gmann at ghanshyammann.com> wrote ----
> > > > > > > Hello Everyone,
> > > > > > >
> > > > > > > We had good amount of discussion on the final plan and schedule in today's TC office hour[1].
> > > > > > >
> > > > > > > I captured the agreement on each point in etherpad (you can see the AGREE:). Also summarizing
> > > > > > > the discussions here. Imp point is if your projects are planning to keep the py2.7 support then do not delay
> > > > > > > to tell us. Reply on this ML thread or add your project in etherpad.
> > > > > > >
> > > > > > > - Projects can start dropping the py2.7 support. Common lib and testing tools need to wait until milestone-2.
> > > > > > > ** pepe8 job to be included in openstack-python3-ussuri-jobs-* templates -
> > > > > > https://review.opendev.org/#/c/688997/
> > > > > > > ** You can drop openstack-python-jobs template and start using ussuri template once 688997 patch is merged.
> > > > > > > ** Cross projects dependency (if any ) can be sync up among dependent projects.
> > > > > > >
> > > > > > > - I will add this plan and schedule as a community goal. The goal is more about what all things to do and when.
> > > > > > > ** If any project keeping the support then it has to be notified explicitly for its consumer.
> > > > > > >
> > > > > > > - Schedule:
> > > > > > > The schedule is aligned with the Ussuri cycle milestone[2]. I will add the plan in the release schedule also.
> > > > > > > Phase-1: Dec 09 - Dec 13 R-22 Ussuri-1 milestone
> > > > > > > ** Project to start dropping the py2 support along with all the py2 CI jobs.
> > > > > > > Phase-2: Feb 10 - Feb 14 R-13 Ussuri-2 milestone
> > > > > > > ** This includes Oslo, QA tools (or any other testing tools), common lib (os-brick), Client library.
> > > > > > > ** This will give enough time to projects to drop the py2 support.
> > > > > > > Phase-3: Apr 06 - Apr 10 R-5 Ussuri-3 milestone
> > > > > > > ** Final audit on Phase-1 and Phase-2 plan and make sure everything is done without breaking anything.
> > > > > > > This is enough time to measure such break or anything extra to do before ussuri final release.
> > > > > > >
> > > > > > > Other discussions points and agreement:
> > > > > > > - Projects want to keep python 2 support and need oslo, QA or any other dependent projects/lib support:
> > > > > > > ** swift. AI: gmann to reach out to swift team about the plan and exact required things from its dependency
> > > > > > > (the common lib/testing tool).
> > > > > >
> > > > > > I chated with timburke on IRC about things required by swift to keep the py2.7 support[1]. Below are
> > > > > > client lib/middleware swift required for py2 testing.
> > > > > > @timburke, feel free to update if any missing point.
> > > > > >
> > > > > > - devstack. able to keep running swift on py2 and rest all services can be on py3
> > > > > > - keystonemiddleware and its dependency
> > > > > > - keystoneclient and openstackclient (dep of keystonemiddleware)
> > > > > > - castellan and barbicanclient
> > > > > >
> > > > > >
> > > > > > As those lib/middleware going to drop the py2.7 support in phase-2, we need to cap them for swift.
> > > > > > I think capping them for python2.7 in upper constraint file would not affect any other users but Matthew Thode can
> > > > > > explain better how that will work from the requirement constraint perspective.
> > > > > >
> > > > > > [1]
> > > > > > http://eavesdrop.openstack.org/irclogs/%23openstack-swift/%23openstack-swift.2019-10-28.log.html#t2019-10-28T16:37:33
> > > > > >
> > > > > > -gmann
> > > > > >
> > > > >
> > > > > ya, there are examples already for libs that have dropped py2 support.
> > > > > What you need to do is update global requirements to be something like
> > > > > the following.
> > > > >
> > > > > sphinx!=1.6.6,!=1.6.7,<2.0.0;python_version=='2.7' # BSD
> > > > > sphinx!=1.6.6,!=1.6.7,!=2.1.0;python_version>='3.4' # BSD
> > > > >
> > > > > or
> > > > >
> > > > > keyring<19.0.0;python_version=='2.7' # MIT/PSF
> > > > > keyring;python_version>='3.4' # MIT/PSF
> > > > on a related note os-vif is blocked form running tempest jobs under python 3
> > > > until https://review.opendev.org/#/c/681029/ is merged due to
> > > > https://zuul.opendev.org/t/openstack/build/4ff60d6bd2f24782abeb12cc7bdb8013/log/controller/logs/screen-q-agt.txt.gz#308-318
> > > >
> > > > i think this issue will affect any job that install proejcts that use privsep using the required-proejcts section of the
> > > > zuul job definition. adding a project to required-proejcts sechtion adds it to the LIBS_FROM_GIT varible in devstack.
> > > > this inturn istalls it twice due to https://review.opendev.org/#/c/418135/ . the side effect of this is that the
> > > > privsep helper script gets installed under python2 and the neutron ageint in this case gets install under python 3 so
> > > > when it trys to spawn the privsep deamon and invoke commands it typically expodes due to dependcy issues or in this case
> > > > because it failed to drop privileges correctly.
> > > >
> > > > so as part of phase 1 we need to merge https://review.opendev.org/#/c/681029/ so that lib project that use required-
> > > > projects to run with master of project that comsume it and support depends-on can move to python 3 tempest jobs.
> > >
> > > Thanks for raising this. I agree on not falling back to py2 in Ussuri, I approved 681029.
> > >
> > > -gmann
> > >
> > > >
> > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
>
More information about the openstack-discuss
mailing list