[openstack-dev] [requirements][infra] Maintaining constraints for several python versions

Monty Taylor mordred at inaugust.com
Thu Jul 12 16:31:34 UTC 2018


On 07/12/2018 08:37 AM, Clark Boylan wrote:
> On Wed, Jul 11, 2018, at 9:34 PM, Tony Breeds wrote:
>> Hi Folks,
>>      We have a pit of a problem in openstack/requirements and I'd liek to
>> chat about it.
>>
>> Currently when we generate constraints we create a venv for each
>> (system) python supplied on the command line, install all of
>> global-requirements into that venv and capture the pip freeze.
>>
>> Where this falls down is if we want to generate a freeze for python 3.4
>> and 3.5 we need an image that has both of those.  We cheated and just
>> 'clone' them so if python3 is 3.4 we copy the results to 3.5 and vice
>> versa.  This kinda worked for a while but it has drawbacks.
>>
>> I can see a few of options:
>>
>> 1. Build pythons from source and use that to construct the venv
>>     [please no]
> 
> Fungi mentions that 3.3 and 3.4 don't build easily on modern linux distros. However, 3.3 and 3.4 are also unsupported by Python at this point, maybe we can ignore them and focus on 3.5 and forward? We don't build new freeze lists for the stable branches, this is just a concern for master right?

FWIW, I use pyenv for python versions on my laptop and love it. I've 
completely given up on distro-provided python for my own usage.

>>
>> 2. Generate the constraints in an F28 image.  My F28 has ample python
>>     versions:
>>       - /usr/bin/python2.6
>>       - /usr/bin/python2.7
>>       - /usr/bin/python3.3
>>       - /usr/bin/python3.4
>>       - /usr/bin/python3.5
>>       - /usr/bin/python3.6
>>       - /usr/bin/python3.7
>>     I don't know how valid this still is but in the past fedora images
>>     have been seen as unstable and hard to keep current.  If that isn't
>>     still the feeling then we could go down this path.  Currently there a
>>     few minor problems with bindep.txt on fedora and generate-constraints
>>     doesn't work with py3 but these are pretty minor really.
> 
> I think most of the problems with Fedora stability are around  bringing up a new Fedora every 6 months or so. They tend to change sufficiently within that time period to make this a fairly involved exercise. But once working they work for the ~13 months of support they offer. I know Paul Belanger would like to iterate more quickly and just keep the most recent Fedora available (rather than ~2).
> 
>>
>> 3. Use docker images for python and generate the constraints with
>>     them.  I've hacked up something we could use as a base for that in:
>>        https://review.openstack.org/581948
>>
>>     There are lots of open questions:
>>       - How do we make this nodepool/cloud provider friendly ?
>>         * Currently the containers just talk to the main debian mirrors.
>>           Do we have debian packages? If so we could just do sed magic.
> 
> http://$MIRROR/debian (http://mirror.dfw.rax.openstack.org/debian for example) should be a working amd64 debian package mirror.
> 
>>       - Do/Can we run a registry per provider?
> 
> We do not, but we do have a caching dockerhub registry proxy in each region/provider. http://$MIRROR:8081/registry-1.docker if using older docker and http://$MIRROR:8082 for current docker. This was a compromise between caching the Internet and reliability.

there is also

https://review.openstack.org/#/c/580730/

which adds a role to install docker and configure it to use the correct 
registry.

>>       - Can we generate and caches these images and only run pip install -U
>>         g-r to speed up the build
> 
> Between cached upstream python docker images and prebuilt wheels mirrored in every cloud provider region I wonder if this will save a significant amount of time? May be worth starting without this and working from there if it remains slow.
> 
>>       - Are we okay with using docker this way?
> 
> Should be fine, particularly if we are consuming the official Python images.

Agree. python:3.6 and friends are great.

>>
>> I like #2 the most but I wanted to seek wider feedback.
> 
> I think each proposed option should work as long as we understand the limitations each presents. #2 should work fine if we have individuals interested and able to spin up new Fedora images and migrate jobs to that image after releases happen.
> 
> Clark
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 




More information about the OpenStack-dev mailing list