Open Stack

I actually use a local Squid proxy for the same thing.  I wonder if the
built-in pip cache would be faster though.  I'll have to give it a shot
sometime.

Another thing I've found helps with subsequent venv rebuilds is to install
and use ccache.  These were the approximate build times for the venv on
one
of my machines:

First:
real    10m30.287s
user    3m35.709s
sys     0m18.445s

Second:
real    5m49.995s
user    0m49.615s
sys     0m12.321s

A pretty nice speedup. :-)

To use ccache you just install it from apt (or your package manager of
choice) then make sure the /usr/lib/ccache directory is in your PATH
before
the actual location of gcc.  Helps a lot with those C libs that don't
change but need to be recompiled every time.

-Ben

On Mon, 13 May 2013 09:34:21 +1000, Matthew Sherborne
<msherborne at gmail.com> wrote:
> Until the wheel thing is available, this might help in making .venv
> creation quicker:
> 
> mkdir -p ~/.pip/cache
> echo export PIP_DOWNLOAD_CACHE=~/.pip/cache >> ~/.bashrc
> 
> Then it makes a nice cache and every time you create a new .venv it
doesn't
> need to download again.
> 
> Kind Regards,
> Matthew Sherborne
> 
> 
> 
> 
> 
> On Tue, May 7, 2013 at 1:51 AM, Monty Taylor <mordred at inaugust.com>
wrote:
> 
>>
>>
>> On 05/04/2013 07:59 PM, Matthew Sherborne wrote:
>> > So I tried to speed up the running of my test suite and failed.
>> >
>> > I thought I'd share my experience in case someone else in the future
>> > googles before
>> >  trying the same thing, or in case some genius out there sees some
>> >  fatal
>> > flaw in my plan.
>> >
>> > Here's what I did:
>> >
>> > I'm on Ubuntu Raring. I have both gcc and clang compilers installed.
>> >
>> > I did some experiments with a c++ test program and found that clang
>> > with
>> > the -O3 optimization flag creates a faster program than gcc with -O6.
>> >
>> > So I thought, I'll compile my own python and venv with python and
clang
>> > and -O3.
>> >
>> > So I compiled my python from source with clang and -O3 and installed
it
>> > in ~/python.
>> >
>> > I deleted my venv for nova and, rebuilt it all under the new python.
>> >
>> > Here are the *results* of: time ./run_tests.sh
>> >
>> > *standard python:*
>> >
>> > real    7m4.077s
>> > user    21m0.080s
>> > sys 0m24.596s
>> >
>> > *with clang compiled python:*
>> >
>> > real    8m11.396s
>> > user    24m28.824s
>> > sys 0m25.408s
>> >
>> > I tried it a few times and got similar results. So it seems that the
>> > stock ubuntu python is faster than my cmake -O3 compiled one.
>> >
>> > All comments and suggestions welcomed.
>>
>> Nice experiment! I do love seeing actual numbers.
>>
>> BTW - we've got a back-burner project to add support for the new python
>> wheel file format to our pypi mirror, and then to build wheels of
things
>> when we pull them down from upstream. While this will do nothing to
>> speed up the actual test runs, it should make virtualenv creation WAY
>> quicker, since we won't have to continually re-compile libxml...
>>
>> thanks for experimenting with that!
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>