Open Stack

On Mon, Jul 14, 2014 at 2:58 AM, Monty Taylor <mordred at inaugust.com> wrote:

> On 07/10/2014 02:44 PM, Richard Jones wrote:
> > On 10 July 2014 23:27, Mulcahy, Stephen <stephen.mulcahy at hp.com> wrote:
> >> When I last tested bandersnatch, it didn’t work well behind a proxy (in
> > fact most of the existing pypi mirroring tools suffered from the same
> > problem) – pypi-mirror has worked extremely well for mirroring a subset
> of
> > pypi and doing so behind a proxy. I’d also echo the requirement for a
> tool
> > that provides wheels as we have seen significant performance improvement
> > from using wheels with TripleO
> >
> > devpi works behind a proxy. If bandersnatch doesn't then that bug should
> be
> > addressed ASAP. I'm in contact with its author regarding that.
> >
> > I'm currently investigating a sensible approach to having wheels be
> > automatically built (for the most sensible value of "automatic" that we
> can
> > determine <wink>).
>
> We're also thinking about how we continue to offer the pre-built wheels
> for each of our build platforms. For infra, what I'm thinking is:
>
> On each mirror slave (We have one for each OS combo we use), do
> something similar to:
>
> pip wheel -r global-requirements.txt
> rsync $wheelhouse pypi.openstack.org/$(lsb_release)
>
> This may require keeping pypi-mirror and using an option to only do
> wheel building so that we can get the directory publication split. Ok. I
> got bored and wrote that:
>
> https://review.openstack.org/106638
>
> So if we land that, you can do;
>
> pip wheel -r global-requirements.txt
> run-mirror --wheels-only --wheelhouse=wheelhouse --wheeldest=mirror
> rsync -avz mirror pypi.openstack.org:/srv/mirror
>
> If we went the devpi route, we could do;
>
> pip wheel -r global-requirements.txt
> for pkg in $wheelhouse; do
>   devpi upload $pkg
> done
>
> And put that into a cron.


Obviously "keeping pypi-mirror" would require the least amount of change to
how we suggest developers set up their systems.

I think the devpi option seems fairly reasonable too. It looks like it's
easier (and faster, and less bandwidth-consuming) than setting up
bandersnatch or apt-mirror, which we currently suggest people consider. It
doesn't look any more heavyweight than having a squid proxy for caching,
which we currently suggest as a bare minimum.

For an individual dev testing their own setup, I think we need a slightly
different approach from the infra approach listed above though. I'm
assuming that it's possible to probe the package index to determine if a
wheel is available for a particular version of a package yet. If that's the
case, we should be able to tweak tools like os-svc-install to notice when
no wheel is available, and build and upload the wheel.

I think this should give us a good balance between making sure that each
build (except the first) uses wheels to save time, still gets the latest
packages (since the last time the system was online at least), and the user
doesn't need to remember to manually update the wheels when they're online.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140714/459f9768/attachment.html>