Open Stack

Excerpts from Derek Higgins's message of 2015-01-07 02:51:41 -0800:
> Hi All,
>     I intended to bring this up at this mornings meeting but the train I
> was on had no power sockets (and I had no battery) so sending to the
> list instead.
> 
> We currently run our CI with on images built for i386, we took this
> decision a while back to save memory ( at the time is allowed us to move
> the amount of memory required in our VMs from 4G to 2G (exactly where in
> those bands the hard requirements are I don't know)
> 
> Since then we have had to move back to 3G for the i386 VM as 2G was no
> longer enough so the saving in memory is no longer as dramatic.
> 
> Now that the difference isn't as dramatic, I propose we switch back to
> amd64 (with 4G vms) in order to CI on what would be closer to a
> production deployment and before making the switch wanted to throw the
> idea out there for others to digest.
> 
> This obviously would impact our capacity as we will have to reduce the
> number of testenvs per testenv hosts. Our capacity (in RH1 and roughly
> speaking) allows us to run about 1440 ci jobs per day. I believe we can
> make the switch and still keep capacity above 1200 with a few other changes
> 1. Add some more testenv hosts, we have 2 unused hosts at the moment and
> we can probably take 2 of the compute nodes from the overcloud.
> 2. Kill VM's at the end of each CI test (as opposed to leaving them
> running until the next CI test kills them), allowing us to more
> successfully overcommit on RAM
> 3. maybe look into adding swap on the test env hosts, they don't
> currently have any, so over committing RAM is a problem the the OOM
> killer is handling from time to time (I only noticed this yesterday).
> 
> The other benefit to doing this is that is we were to ever want to CI
> images build with packages (this has come up in previous meetings) we
> wouldn't need to provide i386 packages just for CI, while the rest of
> the world uses the amd64.

+1 on all counts.

It's also important to note that we should actually have a whole new
rack of servers added to capacity soon (I think soon is about 6 months
so far, but we are at least committed to it). So this would be, at worst,
a temporary loss of 240 jobs per day.