Open Stack

On Sun, Jul 13, 2014, at 02:36 PM, James Polley wrote:


We're also thinking about how we continue to offer the
pre-built wheels

for each of our build platforms. For infra, what I'm thinking
is:



On each mirror slave (We have one for each OS combo we use), do

something similar to:



pip wheel -r global-requirements.txt

rsync $wheelhouse [1]pypi.openstack.org/$(lsb_release)



This may require keeping pypi-mirror and using an option to
only do

wheel building so that we can get the directory publication
split. Ok. I

got bored and wrote that:



[2]https://review.openstack.org/106638



So if we land that, you can do;



pip wheel -r global-requirements.txt

run-mirror --wheels-only --wheelhouse=wheelhouse
--wheeldest=mirror

rsync -avz mirror pypi.openstack.org:/srv/mirror



If we went the devpi route, we could do;



pip wheel -r global-requirements.txt

for pkg in $wheelhouse; do

  devpi upload $pkg

done



And put that into a cron.


Obviously "keeping pypi-mirror" would require the least amount
of change to how we suggest developers set up their systems.

I think the devpi option seems fairly reasonable too. It looks
like it's easier (and faster, and less bandwidth-consuming)
than setting up bandersnatch or apt-mirror, which we currently
suggest people consider. It doesn't look any more heavyweight
than having a squid proxy for caching, which we currently
suggest as a bare minimum.

For an individual dev testing their own setup, I think we need
a slightly different approach from the infra approach listed
above though. I'm assuming that it's possible to probe the
package index to determine if a wheel is available for a
particular version of a package yet. If that's the case, we
should be able to tweak tools like os-svc-install to notice
when no wheel is available, and build and upload the wheel.

I think this should give us a good balance between making sure
that each build (except the first) uses wheels to save time,
still gets the latest packages (since the last time the system
was online at least), and the user doesn't need to remember to
manually update the wheels when they're online.



This gave me an idea:

There was talk about pip being able to use a wheel cache
(wheelhouse). Can we bind-mount an arch-specific wheelhouse
from the hypervisor into our chroots as we build? This would
let people get most of the wheel speedup while doing almost no
specifal configuration.



-Greg

References

1. http://pypi.openstack.org/$(lsb_release)
2. https://review.openstack.org/106638
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140713/890940c0/attachment.html>