Open Stack

Tuomas,

Have you published your code/blueprints anywhere? Looks like we’re working on the same stuff. I have implemented almost the same feature set (haven’t published anything yet because of this thread), except for the scheduler part. The main goal is to be able to pin VCPUs in NUMA environment.

Have you considered adding placement and cpuset attributes to <vcpu> element? For example:
<vcpu placement=‘static’ cpuset=‘%whatever%’>

Thanks,
Roman

On Nov 13, 2013, at 14:46, Tuomas Paappanen <tuomas.paappanen at tieto.com> wrote:

> Hi all,
> 
> I would like to hear your thoughts about core pinning in Openstack. Currently nova(with qemu-kvm) supports usage of cpu set of PCPUs what can be used by instances. I didn't find blueprint, but I think this feature is for isolate cpus used by host from cpus used by instances(VCPUs).
> 
> But, from performance point of view it is better to exclusively dedicate PCPUs for VCPUs and emulator. In some cases you may want to guarantee that only one instance(and its VCPUs) is using certain PCPUs.  By using core pinning you can optimize instance performance based on e.g. cache sharing, NUMA topology, interrupt handling, pci pass through(SR-IOV) in multi socket hosts etc.
> 
> We have already implemented feature like this(PoC with limitations) to Nova Grizzly version and would like to hear your opinion about it.
> 
> The current implementation consists of three main parts:
> - Definition of pcpu-vcpu maps for instances and instance spawning
> - (optional) Compute resource and capability advertising including free pcpus and NUMA topology.
> - (optional) Scheduling based on free cpus and NUMA topology.
> 
> The implementation is quite simple:
> 
> (additional/optional parts)
> Nova-computes are advertising free pcpus and NUMA topology in same manner than host capabilities. Instances are scheduled based on this information.
> 
> (core pinning)
> admin can set PCPUs for VCPUs and for emulator process, or select NUMA cell for instance vcpus, by adding key:value pairs to flavor's extra specs.
> 
> EXAMPLE:
> instance has 4 vcpus
> <key>:<value>
> vcpus:1,2,3,4 --> vcpu0 pinned to pcpu1, vcpu1 pinned to pcpu2...
> emulator:5 --> emulator pinned to pcpu5
> or
> numacell:0 --> all vcpus are pinned to pcpus in numa cell 0.
> 
> In nova-compute, core pinning information is read from extra specs and added to domain xml same way as cpu quota values(cputune).
> 
> <cputune>
>      <vcpupin vcpu='0' cpuset='1'/>
>      <vcpupin vcpu='1' cpuset='2'/>
>      <vcpupin vcpu='2' cpuset='3'/>
>      <vcpupin vcpu='3' cpuset='4'/>
>      <emulatorpin cpuset='5'/>
> </cputune>
> 
> What do you think? Implementation alternatives? Is this worth of blueprint? All related comments are welcome!
> 
> Regards,
> Tuomas
> 
> 
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev