Open Stack

Excerpts from Matthew Treinish's message of 2018-05-29 16:16:50 -0400:
> On Tue, May 29, 2018 at 03:51:31PM -0400, Doug Hellmann wrote:
> > Following up on this topic, at the Forum discussion last week (see
> > https://etherpad.openstack.org/p/YVR-python-2-deprecation-timeline) the
> > general plan outlined below was acceptable to most of the folks in the
> > room with a few small changes (included below).
> > 
> > Excerpts from Doug Hellmann's message of 2018-04-25 16:54:46 -0400:
> > > It's time to talk about the next steps in our migration from python
> > > 2 to python 3.
> > > 
> > > Up to this point we have mostly focused on reaching a state where
> > > we support both versions of the language. We are not quite there
> > > with all projects, as you can see by reviewing the test coverage
> > > status information at
> > > https://wiki.openstack.org/wiki/Python3#Python_3_Status_of_OpenStack_projects
> > > 
> > > Still, we need to press on to the next phase of the migration, which
> > > I have been calling "Python 3 first". This is where we use python
> > > 3 as the default, for everything, and set up the exceptions we need
> > > for anything that still requires python 2.
> > > 
> > > To reach that stage, we need to:
> > > 
> > > 1. Change the documentation and release notes jobs to use python 3.
> > >    (The Oslo team recently completed this, and found that we did
> > >    need to make a few small code changes to get them to work.)
> > > 2. Change (or duplicate) all functional test jobs to run under
> > >    python 3.
> > > 3. Change the packaging jobs to use python 3.
> > > 4. Update devstack to use 3 by default and require setting a flag to
> > >    use 2. (This may trigger other job changes.)
> > 
> > Also:
> > 
> > - Ensure that devstack configures mod_wsgi (or whatever WSGI service) to
> >   use Python 3 when deploying API components.
> 
> The python 3 dsvm jobs already do this for the most part. All API services that
> support running as a wsgi application run under uwsgi with a single apache
> redirecting traffic to those. This is the supported model for running wsgi
> services on devstack. Currently keystone, glance, nova, placement, and cinder
> run their API servers this way. Neutron doesn't run under as wsgi app (I
> don't recall why this was never implemented for neutron) and swift doesn't run
> in the py3 jobs at all. You can see an example of this here:

OK, good, thank you for clarifying that. I think the folks in the room
weren't 100% sure, so the point was to double check. And it sounds like
we still need to do that for projects using devstack plugins, based on
your next comment.

> 
> http://logs.openstack.org/08/550108/3/gate/tempest-full-py3/df744ef/controller/logs/
> 
> For other services it depends on how they implemented their devstack plugin. I
> haven't done an inventory on how all the plugins are running things, so I don't
> know what the status of each project is there.
> 
> > - Test "python version skew" within a service during a rolling upgrade
> >   across multiple hosts.
> > - Add an integration test job that does not include python2 on the host
> >   at all.
> > 
> > That last item may block us from using other tools, such as ansible,
> > that rely on python2. If the point of such a test is to ensure that
> > we are properly installing (and running) our tools under python3,
> > maybe *that's* what we want to check, instead of forbidding a python2
> > package at all? Could we, for example, look at the set of packages
> > installed under python2 and report errors if any OpenStack packages end
> > up there?
> > 
> > > 
> > > At that point, all of our deliverables will be produced using python
> > > 3, and we can be relatively confident that if we no longer had
> > > access to python 2 we could still continue operating. We could also
> > > start updating deployment tools to use either python 3 or 2, so
> > > that users could actually deploy using the python 3 versions of
> > > services.
> > > 
> > > Somewhere in that time frame our third-party CI systems will need
> > > to ensure they have python 3 support as well.
> > > 
> > > After the "Python 3 first" phase is completed we should release
> > > one series using the packages built with python 3. Perhaps Stein?
> > > Or is that too ambitious?
> > > 
> > > Next, we will be ready to address the prerequisites for "Python 3
> > > only," which will allow us to drop Python 2 support.
> > > 
> > > We need to wait to drop python 2 support as a community, rather
> > > than going one project at a time, to avoid doubling the work of
> > > downstream consumers such as distros and independent deployers. We
> > > don't want them to have to package all (or even a large number) of
> > > the dependencies of OpenStack twice because they have to install
> > > some services running under python 2 and others under 3. Ideally
> > > they would be able to upgrade all of the services on a node together
> > > as part of their transition to the new version, without ending up
> > > with a python 2 version of a dependency along side a python 3 version
> > > of the same package.
> > > 
> > > The remaining items could be fixed earlier, but this is the point
> > > at which they would block us:
> > > 
> > > 1. Fix oslo.service functional tests -- the Oslo team needs help
> > >    maintaining this library. Alternatively, we could move all
> > >    services to use cotyledon (https://pypi.org/project/cotyledon/).
> > > 
> > > 2. Finish the unit test and functional test ports so that all of
> > >    our tests can run under python 3 (this implies that the services
> > >    all run under python 3, so there is no more porting to do).
> > > 
> > > Finally, after we have *all* tests running on python 3, we can
> > > safely drop python 2.
> > 
> > We clarified that we would only drop python 2 support on master.
> > That clarification also raised the point that eventually backports
> > may become more difficult if master is using python 3 features, but
> > Matt and Tony agreed that we could potentially have rebasing issues
> > with fixes today so while this is a new source of such issues it
> > isn't a completely new problem and our stable backport policies
> > already address it.
> > 
> > > 
> > > We have previously discussed the end of the T cycle as the point
> > > at which we would have all of those tests running, and if that holds
> > > true we could reasonably drop python 2 during the beginning of the
> > > U cycle, in late 2019 and before the 2020 cut-off point when upstream
> > > python 2 support will be dropped.
> > 
> > This date as the earliest point at which existing projects could drop
> > python 2 seems to be generally acceptable to everyone. I wrote up
> > a TC resolution as a more formal documentation for it:
> > 
> >   https://review.openstack.org/571011
> > 
> > I also intend to propose a "python 3 first" goal for Stein, but I
> > consider those two things independent.
> > 
> > The last point of significant interest is that we discussed modifying
> > Graham's existing governance change to indicate that new projects
> > do not need to have python 2 support with the caveat that platforms
> > that do not support python 3 fully, yet, are unlikely to package
> > those projects. Graham was going to update the patch, IIRC.
> > 
> > > 
> > > I need some info from the deployment tool teams to understand whether
> > > they would be ready to take the plunge during T or U and start
> > > deploying only the python 3 version. Are there other upgrade issues
> > > that need to be addressed to support moving from 2 to 3? Something
> > > that might be part of the platform(s), rather than OpenStack itself?
> > > 
> > > What else have I missed in these phases? Other jobs? Other blocking
> > > conditions?
> > > 
> > > Doug
> > 
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev