Open Stack

Following up on this topic, at the Forum discussion last week (see
https://etherpad.openstack.org/p/YVR-python-2-deprecation-timeline) the
general plan outlined below was acceptable to most of the folks in the
room with a few small changes (included below).

Excerpts from Doug Hellmann's message of 2018-04-25 16:54:46 -0400:
> It's time to talk about the next steps in our migration from python
> 2 to python 3.
> 
> Up to this point we have mostly focused on reaching a state where
> we support both versions of the language. We are not quite there
> with all projects, as you can see by reviewing the test coverage
> status information at
> https://wiki.openstack.org/wiki/Python3#Python_3_Status_of_OpenStack_projects
> 
> Still, we need to press on to the next phase of the migration, which
> I have been calling "Python 3 first". This is where we use python
> 3 as the default, for everything, and set up the exceptions we need
> for anything that still requires python 2.
> 
> To reach that stage, we need to:
> 
> 1. Change the documentation and release notes jobs to use python 3.
>    (The Oslo team recently completed this, and found that we did
>    need to make a few small code changes to get them to work.)
> 2. Change (or duplicate) all functional test jobs to run under
>    python 3.
> 3. Change the packaging jobs to use python 3.
> 4. Update devstack to use 3 by default and require setting a flag to
>    use 2. (This may trigger other job changes.)

Also:

- Ensure that devstack configures mod_wsgi (or whatever WSGI service) to
  use Python 3 when deploying API components.
- Test "python version skew" within a service during a rolling upgrade
  across multiple hosts.
- Add an integration test job that does not include python2 on the host
  at all.

That last item may block us from using other tools, such as ansible,
that rely on python2. If the point of such a test is to ensure that
we are properly installing (and running) our tools under python3,
maybe *that's* what we want to check, instead of forbidding a python2
package at all? Could we, for example, look at the set of packages
installed under python2 and report errors if any OpenStack packages end
up there?

> 
> At that point, all of our deliverables will be produced using python
> 3, and we can be relatively confident that if we no longer had
> access to python 2 we could still continue operating. We could also
> start updating deployment tools to use either python 3 or 2, so
> that users could actually deploy using the python 3 versions of
> services.
> 
> Somewhere in that time frame our third-party CI systems will need
> to ensure they have python 3 support as well.
> 
> After the "Python 3 first" phase is completed we should release
> one series using the packages built with python 3. Perhaps Stein?
> Or is that too ambitious?
> 
> Next, we will be ready to address the prerequisites for "Python 3
> only," which will allow us to drop Python 2 support.
> 
> We need to wait to drop python 2 support as a community, rather
> than going one project at a time, to avoid doubling the work of
> downstream consumers such as distros and independent deployers. We
> don't want them to have to package all (or even a large number) of
> the dependencies of OpenStack twice because they have to install
> some services running under python 2 and others under 3. Ideally
> they would be able to upgrade all of the services on a node together
> as part of their transition to the new version, without ending up
> with a python 2 version of a dependency along side a python 3 version
> of the same package.
> 
> The remaining items could be fixed earlier, but this is the point
> at which they would block us:
> 
> 1. Fix oslo.service functional tests -- the Oslo team needs help
>    maintaining this library. Alternatively, we could move all
>    services to use cotyledon (https://pypi.org/project/cotyledon/).
> 
> 2. Finish the unit test and functional test ports so that all of
>    our tests can run under python 3 (this implies that the services
>    all run under python 3, so there is no more porting to do).
> 
> Finally, after we have *all* tests running on python 3, we can
> safely drop python 2.

We clarified that we would only drop python 2 support on master.
That clarification also raised the point that eventually backports
may become more difficult if master is using python 3 features, but
Matt and Tony agreed that we could potentially have rebasing issues
with fixes today so while this is a new source of such issues it
isn't a completely new problem and our stable backport policies
already address it.

> 
> We have previously discussed the end of the T cycle as the point
> at which we would have all of those tests running, and if that holds
> true we could reasonably drop python 2 during the beginning of the
> U cycle, in late 2019 and before the 2020 cut-off point when upstream
> python 2 support will be dropped.

This date as the earliest point at which existing projects could drop
python 2 seems to be generally acceptable to everyone. I wrote up
a TC resolution as a more formal documentation for it:

  https://review.openstack.org/571011

I also intend to propose a "python 3 first" goal for Stein, but I
consider those two things independent.

The last point of significant interest is that we discussed modifying
Graham's existing governance change to indicate that new projects
do not need to have python 2 support with the caveat that platforms
that do not support python 3 fully, yet, are unlikely to package
those projects. Graham was going to update the patch, IIRC.

> 
> I need some info from the deployment tool teams to understand whether
> they would be ready to take the plunge during T or U and start
> deploying only the python 3 version. Are there other upgrade issues
> that need to be addressed to support moving from 2 to 3? Something
> that might be part of the platform(s), rather than OpenStack itself?
> 
> What else have I missed in these phases? Other jobs? Other blocking
> conditions?
> 
> Doug