Open Stack

On Sep 3, 2014, at 11:57 AM, Clark Boylan <cboylan at sapwetik.org> wrote:

> On Wed, Sep 3, 2014, at 08:22 AM, Doug Hellmann wrote:
>> 
>> On Sep 2, 2014, at 3:17 PM, Clark Boylan <cboylan at sapwetik.org> wrote:
>> 
>>> On Tue, Sep 2, 2014, at 11:30 AM, Yuriy Taraday wrote:
>>>> Hello.
>>>> 
>>>> Currently for alpha releases of oslo libraries we generate either
>>>> universal
>>>> or Python 2.x-only wheels. This presents a problem: we can't adopt alpha
>>>> releases in projects where Python 3.x is supported and verified in the
>>>> gate. I've ran into this in change request [1] generated after
>>>> global-requirements change [2]. There we have oslotest library that can't
>>>> be built as a universal wheel because of different requirements (mox vs
>>>> mox3 as I understand is the main difference). Because of that py33 job in
>>>> [1] failed and we can't bump oslotest version in requirements.
>>>> 
>>>> I propose to change infra scripts that generate and upload wheels to
>>>> create
>>>> py3 wheels as well as py2 wheels for projects that support Python 3.x (we
>>>> can use setup.cfg classifiers to find that out) but don't support
>>>> universal
>>>> wheels. What do you think about that?
>>>> 
>>>> [1] https://review.openstack.org/117940
>>>> [2] https://review.openstack.org/115643
>>>> 
>>>> -- 
>>>> 
>>>> Kind regards, Yuriy.
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> We may find that we will need to have py3k wheels in addition to the
>>> existing wheels at some point, but I don't think this use case requires
>>> it. If oslo.test needs to support python2 and python3 it should use mox3
>>> in both cases which claims to support python2.6, 2.7 and 3.2. Then you
>>> can ship a universal wheel. This should solve the immediate problem.
>> 
>> That sounds like a good solution to that specific issue. It may also
>> require changes in the application test suites, but those changes can be
>> made as we move them to use oslotest.
>> 
>>> 
>>> It has been pointed out to me that one case where it won't be so easy is
>>> oslo.messaging and its use of eventlet under python2. Messaging will
>>> almost certainly need python 2 and python 3 wheels to be separate. I
>>> think we should continue to use universal wheels where possible and only
>>> build python2 and python3 wheels in the special cases where necessary.
>>> 
>>> The setup.cfg classifiers should be able to do that for us, though PBR
>>> may need updating? We will also need to learn to upload potentially >1
>> 
>> How do you see that working? We want all of the Oslo libraries to,
>> eventually, support both python 2 and 3. How would we use the classifiers
>> to tell when to build a universal wheel and when to build separate
>> wheels?
>> 
> The classifiers provide info on the versions of python we support. By
> default we can build python2 wheel if only 2 is supported, build python3
> wheel if only 3 is supported, build a universal wheel if both are
> supported. Then we can add a setup.cfg flag to override the universal
> wheel default to build both a python2 and python3 wheel instead. Dstufft
> and mordred should probably comment on this idea before we implement
> anything.

OK. I’m not aware of any python-3-only projects, and the flag to override the universal wheel is the piece I was missing. I think there’s already a setuptools flag related to whether or not we should build universal wheels, isn’t there?

>> 
>>> wheel in our wheel jobs. That bit is likely straight foward. The last
>>> thing that we need to make sure we do is that we have some testing in
>>> place for the special wheels. We currently have the requirements
>>> integration test which runs under python2 checking that we can actually
>>> install all the things together. This ends up exercising our wheels and
>>> checking that they actually work. We don't have a python3 equivalent for
>> 
>> We only know the wheels can be installed. We don’t actually have a test
>> that installs our code and runs it any more (devstack uses “develop” mode
>> which bypasses some of the installation steps, as we found while fixing
>> the recent neutron/pbr issue with a missing config file in their
>> packaging instructions).
>> 
> Yup, this would be a second level of testing that we should consider as
> well. The post install can it do stuff test. We moved our unittesting
> away from using sdist installs in preference for setup.py develop
> equivalent which means that our unittests no longer cover some of this
> stuff for us. Sorry if I hijacked this thread into a "how do we test
> release artifacts?" thread.

I wonder if we should have a flag in devstack to control whether we install in develop or “regular” mode? That would let us test real installations, but still have editable versions for local developer systems.

>> 
>>> that job. It may be better to work out some explicit checking of the
>>> wheels we produce that applies to both versions of python. I am not
>>> quite sure how we should approach that yet.
>> 
>> To fix the asymmetric gating we have between pbr and everything else,
>> Robert suggested setting up some sort of job to install pbr and then
>> build and install a package for the project being tested. We already, as
>> you point out, have a job that does this for all of the projects to test
>> changes to pbr itself. Maybe we can run the same test under python 2 and
>> 3 as part of the same job?
>> 
> And to get that extra level of tested we can run the unittests in that
> package install. We won't be able to run everything against python3
> because of missing support (well we can test installs but the second
> level testing will break if a project doesn't support python3 properly)
> so we will need some way of selecting that. Using two jobs may be
> simplest. But that is an implementation detail.

If we’re going to do that, we could also just have the test job remove the instruction in tox.ini to disable creating the sdist and installing the package.

Doug

>> 
>> Doug
>> 
>>> 
>>> Clark
>>> 
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev