Open Stack

On 07/01/15 23:41, Ben Nemec wrote:
> On 01/07/2015 11:29 AM, Clint Byrum wrote:
>> Excerpts from Derek Higgins's message of 2015-01-07 02:51:41 -0800:
>>> Hi All,
>>>     I intended to bring this up at this mornings meeting but the train I
>>> was on had no power sockets (and I had no battery) so sending to the
>>> list instead.
>>>
>>> We currently run our CI with on images built for i386, we took this
>>> decision a while back to save memory ( at the time is allowed us to move
>>> the amount of memory required in our VMs from 4G to 2G (exactly where in
>>> those bands the hard requirements are I don't know)
>>>
>>> Since then we have had to move back to 3G for the i386 VM as 2G was no
>>> longer enough so the saving in memory is no longer as dramatic.
>>>
>>> Now that the difference isn't as dramatic, I propose we switch back to
>>> amd64 (with 4G vms) in order to CI on what would be closer to a
>>> production deployment and before making the switch wanted to throw the
>>> idea out there for others to digest.
>>>
>>> This obviously would impact our capacity as we will have to reduce the
>>> number of testenvs per testenv hosts. Our capacity (in RH1 and roughly
>>> speaking) allows us to run about 1440 ci jobs per day. I believe we can
>>> make the switch and still keep capacity above 1200 with a few other changes
>>> 1. Add some more testenv hosts, we have 2 unused hosts at the moment and
>>> we can probably take 2 of the compute nodes from the overcloud.
>>> 2. Kill VM's at the end of each CI test (as opposed to leaving them
>>> running until the next CI test kills them), allowing us to more
>>> successfully overcommit on RAM
>>> 3. maybe look into adding swap on the test env hosts, they don't
>>> currently have any, so over committing RAM is a problem the the OOM
>>> killer is handling from time to time (I only noticed this yesterday).
>>>
>>> The other benefit to doing this is that is we were to ever want to CI
>>> images build with packages (this has come up in previous meetings) we
>>> wouldn't need to provide i386 packages just for CI, while the rest of
>>> the world uses the amd64.
>>
>> +1 on all counts.
>>
>> It's also important to note that we should actually have a whole new
>> rack of servers added to capacity soon (I think soon is about 6 months
>> so far, but we are at least committed to it). So this would be, at worst,
>> a temporary loss of 240 jobs per day.
> 
> Actually it should be sooner than that - hp1 still isn't in the CI
> rotation yet, so once that infra change merges (the only thing
> preventing us from using it AFAIK) we'll be getting a bunch more
> capacity in the much nearer term.  Unless Derek is already counting that
> in his estimates above, of course.
Yes, this is correct, hp1 isn't in use at the moment and approximately
double those numbers.

> 
> I don't feel like we've been all that capacity constrained lately
> anyway, so as I said in my other (largely unnecessary, as it turns out)
> email, I'm +1 on doing this.
Correct we're not currently constrained on capacity at all (most days we
run less then 300 jobs), but once the other region is in use we'll be
hoping to add jobs to other projects.

> 
> -Ben
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>