Open Stack

The CERN private cloud is now approaching 40,000 cores in 3 cells. There are another 2 OpenStack clouds at CERN with a further 10-20K cores each (all on KVM). We've got about 4,000 VMs (since some of our workload requires a large number of cores per instance). The last majority is running KVM but we've got some Hyper-V also. We run with hyper-threading. The servers are a mixture of 24, 32 and 48 physical core boxes with around 2GB per core.

The details of the scaling approach was described in the summit at http://www.openstack.org/summit/openstack-summit-hong-kong-2013/session-videos/presentation/deep-dive-into-the-cern-cloud-infrastructure

>From my personal advice from our experience with the CERN workload,

- Keep the number of flavors to a reasonably small number... lots of exotic flavors leads to gaps on the hypervisors
- Watch the disk space on the hypervisors if you've got lots of images and lots of VMs
- Don't over-commit memory unless you're really sure that the workload is inactive. Swapping is very expensive.
- The relative differences between Hypervisors in features should be balanced with your experience in running them. Choose the one you know to start with (you can always change later if you need to and run the two in parallel to compare)
- The human side to scaling is also important ... getting procedures for deployment/upgrade/monitoring which are close to O(1) rather than O(n) will make all the difference as you add more resources.

I've noted down some other points in http://openstack-in-production.blogspot.fr/ over the past 6 months since we've been in production on Grizzly.

Tim

> -----Original Message-----
> From: Jonathan Proulx [mailto:jon at jonproulx.com]
> Sent: 09 January 2014 16:20
> To: Mridhul Pax
> Cc: openstack at lists.openstack.org
> Subject: Re: [Openstack] Using KVM for Large Infrastructures
> 
> On Thu, Jan 9, 2014 at 9:17 AM, Mridhul Pax <mridhul at live.com> wrote:
> > Hello Stackers,
> >
> > Any one using KVM as hypervisor in the production servers ? We are
> > planning to use KVM as hypervisor in our production systems and we are
> > planning to use 5 hypervisor nodes (~40 vms per hypervisor).
> 
> I have 60 hypervisor nodes running kvm and consider my site on the small side of medium.
> 
> The number of VMs per hypervisor is a very tricky question, a better though no less tricky question is the ratio of physical CPUs to virtual
> CPUs.  This is much more a function of you expected work load than the hypervisor you choose.
> 
> For compute intensive workloads we schedule 1:1, for random webhosting type loads we've run up to 9:1 and still had the physical systems
> 80% idle.  In practice our mixed development/research environment (which is what we do http://www.csail.mit.edu) seems to run best at
> 3:1 or 4:1. So our physical nodes are dual socket hex core with hyperthreading enabled so they present 24 'cpus' (threads really) to the
> hypervisor and max out with our work loads hosting 72 to 96 vCPUs.
>  We find memory availability is currently our more limiting factor.
> We do keep a separate set of nodes (a host_aggregate in openstack
> terms) scheduled 1:1 for pure compute instances, if those were mixed with the general compute we'd need to significantly reduce our
> allocation ratios
> 
> > How scalable is KVM when used with openstack ? Any limitations ?
> 
> I'd say given opensstack's architecture the hypervisor choice doesn't impact horizontal scale out at all, though I'm only really familiar with
> Xen and KVM but I'd really be surprised if other hypervisors impacted the number of compute nodes you could deploy with them.
> 
> > Any ideas on handling HA (live migration) and Storage Motions ?
> >
> > Is there a way I can restrict IOPS per VMs ?
> 
> I'm not currently doing either of these so I'll leave the answer to others, pretty sure they're both possible
> 
> > Which Hypervisor is recommended to use with Openstack ?
> 
> That's pretty much the same as asking what the best hypervisor is.
> 
> A reasonable answer is "the one you're most familiar with".  A better answer would be to test your workload on a variety of hypervisors
> and see which works best for the load and features you need.  Performance wise they will be the same inside or outside OpenStack.  I
> think most hypervisors have pretty good feature coverage in OpenStack but it's not inherent so if a feature is important you should look in
> the OpenStack docs and see it's exposed.
> 
> Popularity is also a consideration in that more popular choices tend to have better support because more people work on them and you're
> more likely to get responses to questions from the larger user community. http://www.openstack.org/summit/portland-2013/session-
> videos/presentation/openstack-user-committee-update-and-survey-results
> has results of the latest User Survey if you're interested in what's popular (hypervisors are on slide 16, KVM is used by 71% of
> respondents)
> 
> -Jon
> 
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack