Open Stack

On 01/07/2015 11:29 AM, Clint Byrum wrote:
> Excerpts from Derek Higgins's message of 2015-01-07 02:51:41 -0800:
>> Hi All,
>>     I intended to bring this up at this mornings meeting but the train I
>> was on had no power sockets (and I had no battery) so sending to the
>> list instead.
>>
>> We currently run our CI with on images built for i386, we took this
>> decision a while back to save memory ( at the time is allowed us to move
>> the amount of memory required in our VMs from 4G to 2G (exactly where in
>> those bands the hard requirements are I don't know)
>>
>> Since then we have had to move back to 3G for the i386 VM as 2G was no
>> longer enough so the saving in memory is no longer as dramatic.
>>
>> Now that the difference isn't as dramatic, I propose we switch back to
>> amd64 (with 4G vms) in order to CI on what would be closer to a
>> production deployment and before making the switch wanted to throw the
>> idea out there for others to digest.
>>
>> This obviously would impact our capacity as we will have to reduce the
>> number of testenvs per testenv hosts. Our capacity (in RH1 and roughly
>> speaking) allows us to run about 1440 ci jobs per day. I believe we can
>> make the switch and still keep capacity above 1200 with a few other changes
>> 1. Add some more testenv hosts, we have 2 unused hosts at the moment and
>> we can probably take 2 of the compute nodes from the overcloud.
>> 2. Kill VM's at the end of each CI test (as opposed to leaving them
>> running until the next CI test kills them), allowing us to more
>> successfully overcommit on RAM
>> 3. maybe look into adding swap on the test env hosts, they don't
>> currently have any, so over committing RAM is a problem the the OOM
>> killer is handling from time to time (I only noticed this yesterday).
>>
>> The other benefit to doing this is that is we were to ever want to CI
>> images build with packages (this has come up in previous meetings) we
>> wouldn't need to provide i386 packages just for CI, while the rest of
>> the world uses the amd64.
> 
> +1 on all counts.
> 
> It's also important to note that we should actually have a whole new
> rack of servers added to capacity soon (I think soon is about 6 months
> so far, but we are at least committed to it). So this would be, at worst,
> a temporary loss of 240 jobs per day.

Actually it should be sooner than that - hp1 still isn't in the CI
rotation yet, so once that infra change merges (the only thing
preventing us from using it AFAIK) we'll be getting a bunch more
capacity in the much nearer term.  Unless Derek is already counting that
in his estimates above, of course.

I don't feel like we've been all that capacity constrained lately
anyway, so as I said in my other (largely unnecessary, as it turns out)
email, I'm +1 on doing this.

-Ben