Open Stack

On 2014-03-13 11:12, James Slagle wrote:
> On Thu, Mar 13, 2014 at 2:51 AM, Robert Collins
> <robertc at robertcollins.net> wrote:
>> So we already have pretty high requirements - its basically a 16G
>> workstation as minimum.
>> 
>> Specifically to test the full story:
>>  - a seed VM
>>  - an undercloud VM (bm deploy infra)
>>  - 1 overcloud control VM
>>  - 2 overcloud hypervisor VMs
>> ====
>>    5 VMs with 2+G RAM each.
>> 
>> To test the overcloud alone against the seed we save 1 VM, to skip the
>> overcloud we save 3.
>> 
>> However, as HA matures we're about to add 4 more VMs: we need a HA
>> control plane for both the under and overclouds:
>>  - a seed VM
>>  - 3 undercloud VMs (HA bm deploy infra)
>>  - 3 overcloud control VMs (HA)
>>  - 2 overcloud hypervisor VMs
>> ====
>>    9 VMs with 2+G RAM each == 18GB
>> 
>> What should we do about this?
>> 
>> A few thoughts to kick start discussion:
>>  - use Ironic to test across multiple machines (involves tunnelling
>> brbm across machines, fairly easy)
>>  - shrink the VM sizes (causes thrashing)
>>  - tell folk to toughen up and get bigger machines (ahahahahaha, no)
>>  - make the default configuration inline the hypervisors on the
>> overcloud with the control plane:
>>    - a seed VM
>>    - 3 undercloud VMs (HA bm deploy infra)
>>    - 3 overcloud all-in-one VMs (HA)
>>   ====
>>      7 VMs with 2+G RAM each == 14GB
>> 
>> 
>> I think its important that we exercise features like HA and live
>> migration regularly by developers, so I'm quite keen to have a fairly
>> solid systematic answer that will let us catch things like bad
>> firewall rules on the control node preventing network tunnelling
>> etc... e.g. we benefit the more things are split out like scale
>> deployments are. OTOH testing the micro-cloud that folk may start with
>> is also a really good idea....
> 
> 
> The idea I was thinking was to make a testenv host available to
> tripleo atc's. Or, perhaps make it a bit more locked down and only
> available to a new group of tripleo folk, existing somewhere between
> the privileges of tripleo atc's and tripleo-cd-admins.  We could
> document how you use the cloud (Red Hat's or HP's) rack to start up a
> instance to run devtest on one of the compute hosts, request and lock
> yourself a testenv environment on one of the testenv hosts, etc.
> Basically, how our CI works. Although I think we'd want different
> testenv hosts for development vs what runs the CI, and would need to
> make sure everything was locked down appropriately security-wise.
> 
> Some other ideas:
> 
> - Allow an option to get rid of the seed VM, or make it so that you
> can shut it down after the Undercloud is up. This only really gets rid
> of 1 VM though, so it doesn't buy you much nor solve any long term
> problem.
> 
> - Make it easier to see how you'd use virsh against any libvirt host
> you might have lying around.  We already have the setting exposed, but
> make it a bit more public and call it out more in the docs. I've
> actually never tried it myself, but have been meaning to.
> 
> - I'm really reaching now, and this may be entirely unrealistic :),
> but....somehow use the fake baremetal driver and expose a mechanism to
> let the developer specify the already setup undercloud/overcloud
> environment ahead of time.
> For example:
> * Build your undercloud images with the vm element since you won't be
> PXE booting it
> * Upload your images to a public cloud, and boot instances for them.
> * Use this new mechanism when you run devtest (presumably running from
> another instance in the same cloud)  to say "I'm using the fake
> baremetal driver, and here are the  IP's of the undercloud instances".
> * Repeat steps for the overcloud (e.g., configure undercloud to use
> fake baremetal driver, etc).
> * Maybe it's not the fake baremetal driver, and instead a new driver
> that is a noop for the pxe stuff, and the power_on implementation
> powers on the cloud instances.
> * Obviously if your aim is to test the pxe and disk deploy process
> itself, this wouldn't work for you.
> * Presumably said public cloud is OpenStack, so we've also achieved
> another layer of "On OpenStack".

I actually spent quite a while looking into something like this last 
option when I first started on TripleO, because I had only one big 
server locally and it was running my OpenStack installation.  I was 
hoping to use it for my TripleO instances, and even went so far as to 
add support for OpenStack to the virtual power driver in baremetal.  I 
was never completely successful, but I did work through a number of 
problems:

1. Neutron didn't like allowing the DHCP/PXE traffic to let my seed 
serve to the undercloud.  I was able to get around this by using flat 
networking with a local bridge on the OpenStack system, but I'm not sure 
if that's going to be possible on most public cloud providers.  There 
may very well be a less invasive way to configure Neutron to allow that, 
but I don't know how to do it.

2. Last time I checked, Nova doesn't support PXE booting instances so I 
had to use iPXE images to do the booting.  This doesn't work since we 
PXE boot every time an instance reboots and the iPXE image gets 
overwritten by the image deploy, so the instance doesn't boot properly 
after deployment.  This is where I stopped my investigation because I 
didn't want to start hacking up my OpenStack installation to get around 
the problem, but if we decided to go in this direction I don't think it 
would be terribly difficult to get support for this into Nova.  I know 
there have been proposed patches for it before, but I don't think there 
was ever much push behind them because it isn't something most people 
need to do.

So if we can work through a couple of problems then in theory it should 
be possible to use OpenStack instances for TripleO development, which 
would let us do the cloudy thing and have someone else worry about the 
hardware.  The idea certainly has some appeal to me.

-Ben