Open Stack

> On 14 Jul 2014, at 08:08, Gregory Haynes <greg at greghaynes.net> wrote:
> 
>  
>> On Sun, Jul 13, 2014, at 02:36 PM, James Polley wrote:
>>  
>> We're also thinking about how we continue to offer the pre-built wheels
>> for each of our build platforms. For infra, what I'm thinking is:
>>  
>> On each mirror slave (We have one for each OS combo we use), do
>> something similar to:
>>  
>> pip wheel -r global-requirements.txt
>> rsync $wheelhouse pypi.openstack.org/$(lsb_release)
>>  
>> This may require keeping pypi-mirror and using an option to only do
>> wheel building so that we can get the directory publication split. Ok. I
>> got bored and wrote that:
>>  
>> https://review.openstack.org/106638
>>  
>> So if we land that, you can do;
>>  
>> pip wheel -r global-requirements.txt
>> run-mirror --wheels-only --wheelhouse=wheelhouse --wheeldest=mirror
>> rsync -avz mirror pypi.openstack.org:/srv/mirror
>>  
>> If we went the devpi route, we could do;
>>  
>> pip wheel -r global-requirements.txt
>> for pkg in $wheelhouse; do
>>   devpi upload $pkg
>> done
>>  
>> And put that into a cron.
>>  
>> Obviously "keeping pypi-mirror" would require the least amount of change to how we suggest developers set up their systems.
>>  
>> I think the devpi option seems fairly reasonable too. It looks like it's easier (and faster, and less bandwidth-consuming) than setting up bandersnatch or apt-mirror, which we currently suggest people consider. It doesn't look any more heavyweight than having a squid proxy for caching, which we currently suggest as a bare minimum.
>>  
>> For an individual dev testing their own setup, I think we need a slightly different approach from the infra approach listed above though. I'm assuming that it's possible to probe the package index to determine if a wheel is available for a particular version of a package yet. If that's the case, we should be able to tweak tools like os-svc-install to notice when no wheel is available, and build and upload the wheel.
>>  
>> I think this should give us a good balance between making sure that each build (except the first) uses wheels to save time, still gets the latest packages (since the last time the system was online at least), and the user doesn't need to remember to manually update the wheels when they're online.
>  
> This gave me an idea:
> There was talk about pip being able to use a wheel cache (wheelhouse). Can we bind-mount an arch-specific wheelhouse from the hypervisor into our chroots as we build? This would let people get most of the wheel speedup while doing almost no specifal configuration.

Pip still doesn't handle case insensitivity on file:// indexurls as well as its does for http:// - that should be fixed with a 1.6.0 release. There's some chance that we'll run into issues with this that we wouldn't hit using http, but i don't expect any major issues.

This still leaves us needing to build that arch-specific wheelhouse though


>  
> -Greg
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140714/d39d809d/attachment.html>