Open Stack

On 07/10/2014 02:44 PM, Richard Jones wrote:
> On 10 July 2014 23:27, Mulcahy, Stephen <stephen.mulcahy at hp.com> wrote:
>> When I last tested bandersnatch, it didn’t work well behind a proxy (in
> fact most of the existing pypi mirroring tools suffered from the same
> problem) – pypi-mirror has worked extremely well for mirroring a subset of
> pypi and doing so behind a proxy. I’d also echo the requirement for a tool
> that provides wheels as we have seen significant performance improvement
> from using wheels with TripleO
> 
> devpi works behind a proxy. If bandersnatch doesn't then that bug should be
> addressed ASAP. I'm in contact with its author regarding that.
> 
> I'm currently investigating a sensible approach to having wheels be
> automatically built (for the most sensible value of "automatic" that we can
> determine <wink>).

We're also thinking about how we continue to offer the pre-built wheels
for each of our build platforms. For infra, what I'm thinking is:

On each mirror slave (We have one for each OS combo we use), do
something similar to:

pip wheel -r global-requirements.txt
rsync $wheelhouse pypi.openstack.org/$(lsb_release)

This may require keeping pypi-mirror and using an option to only do
wheel building so that we can get the directory publication split. Ok. I
got bored and wrote that:

https://review.openstack.org/106638

So if we land that, you can do;

pip wheel -r global-requirements.txt
run-mirror --wheels-only --wheelhouse=wheelhouse --wheeldest=mirror
rsync -avz mirror pypi.openstack.org:/srv/mirror

If we went the devpi route, we could do;

pip wheel -r global-requirements.txt
for pkg in $wheelhouse; do
  devpi upload $pkg
done

And put that into a cron.