[openstack-dev] [requirements][infra] Maintaining constraints for several python versions

Tony Breeds tony at bakeyournoodle.com
Thu Jul 12 23:54:09 UTC 2018


On Thu, Jul 12, 2018 at 06:37:52AM -0700, Clark Boylan wrote:
> On Wed, Jul 11, 2018, at 9:34 PM, Tony Breeds wrote:
> > 1. Build pythons from source and use that to construct the venv
> >    [please no]
> 
> Fungi mentions that 3.3 and 3.4 don't build easily on modern linux distros. However, 3.3 and 3.4 are also unsupported by Python at this point, maybe we can ignore them and focus on 3.5 and forward? We don't build new freeze lists for the stable branches, this is just a concern for master right?

The focus is master, but it came up in the context of shoudl we just
remove the python_version=='3.4', it turns out that at least one OS that
will supported rock will be running with python 3.4 so while 3.4 is EOL
I have to admit I'd quite like to be able to keep the 3.4 stuff around
for rocky (and probably stein).

It isn't a hard requirement.

> > 2. Generate the constraints in an F28 image.  My F28 has ample python
> >    versions:
> >      - /usr/bin/python2.6
> >      - /usr/bin/python2.7
> >      - /usr/bin/python3.3
> >      - /usr/bin/python3.4
> >      - /usr/bin/python3.5
> >      - /usr/bin/python3.6
> >      - /usr/bin/python3.7
> >    I don't know how valid this still is but in the past fedora images
> >    have been seen as unstable and hard to keep current.  If that isn't
> >    still the feeling then we could go down this path.  Currently there a
> >    few minor problems with bindep.txt on fedora and generate-constraints
> >    doesn't work with py3 but these are pretty minor really.
> 
> I think most of the problems with Fedora stability are around  bringing up a new Fedora every 6 months or so. They tend to change sufficiently within that time period to make this a fairly involved exercise. But once working they work for the ~13 months of support they offer. I know Paul Belanger would like to iterate more quickly and just keep the most recent Fedora available (rather than ~2).

Ok that's good context.  It isn't that once the images are built they
break it that they're hardish to build in the first place.  I'd love to
think that between Paul, Ian and I we'd be okay here but then again I
don't really know what I'm saying ;P

> > 3. Use docker images for python and generate the constraints with
> >    them.  I've hacked up something we could use as a base for that in:
> >       https://review.openstack.org/581948
> > 
> >    There are lots of open questions:
> >      - How do we make this nodepool/cloud provider friendly ?
> >        * Currently the containers just talk to the main debian mirrors.
> >          Do we have debian packages? If so we could just do sed magic.
> 
> http://$MIRROR/debian (http://mirror.dfw.rax.openstack.org/debian for example) should be a working amd64 debian package mirror.

\o/
 
> >      - Do/Can we run a registry per provider?
> 
> We do not, but we do have a caching dockerhub registry proxy in each region/provider. http://$MIRROR:8081/registry-1.docker if using older docker and http://$MIRROR:8082 for current docker. This was a compromise between caching the Internet and reliability.

That'll do as long as it's easy to configure or transparent.
 
> >      - Can we generate and caches these images and only run pip install -U
> >        g-r to speed up the build
> 
> Between cached upstream python docker images and prebuilt wheels mirrored in every cloud provider region I wonder if this will save a significant amount of time? May be worth starting without this and working from there if it remains slow.

Yeah it may be that I'm over thinking it.  For me (locally) it's really
slow but perhaps with infrastructure you've mentioned it isn't worth it.
Certainly something to look at later if it's a problem.

> >      - Are we okay with using docker this way?
> 
> Should be fine, particularly if we are consuming the official Python images.

Yup that's the plan.  I've sent a PR to get some images we'd need built
that aren't there today.
> 
> > 
> > I like #2 the most but I wanted to seek wider feedback.
> 
> I think each proposed option should work as long as we understand the limitations each presents. #2 should work fine if we have individuals interested and able to spin up new Fedora images and migrate jobs to that image after releases happen.

Yours Tony.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180713/e530ff56/attachment.sig>


More information about the OpenStack-dev mailing list