<div dir="ltr"><div><div class="gmail_extra">@Jay, actually I'm here for CAT. I also have another idea about the proposal, so catch you about it, let us sync all the ideas. :)</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks</div><div class="gmail_extra">Alex</div><div class="gmail_extra"><br><div class="gmail_quote">2017-02-22 11:17 GMT-05:00 Jay Pipes <span dir="ltr"><<a href="mailto:jaypipes@gmail.com" target="_blank">jaypipes@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Eli,<br>
<br>
Sorry for top-posting. Just a quick note to say I had a good conversation on Monday about this with Sean Mooney. I think we have some ideas on how to model all of these resources in the new placement/resource providers schema.<br>
<br>
Are you at the PTG? If so, would be great to meet up to discuss...<br>
<br>
Best,<br>
-jay<span class=""><br>
<br>
On 02/21/2017 05:38 AM, Qiao, Liyong wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
Hi folks:<br>
<br>
<br>
<br>
Seeking community input on an initial design for Intel Resource Director<br>
Technology (RDT), in particular for Cache Allocation Technology in<br>
OpenStack Nova to protect workloads from co-resident noisy neighbors, to<br>
ensure quality of service (QoS).<br>
<br>
<br>
<br></span>
1. What is Cache Allocation Technology (CAT)?**<span class=""><br>
<br>
Intel’s RDT(Resource Director Technology) [1] is a umbrella of<br></span>
*hardware* support to facilitate the monitoring and reservation of<span class=""><br>
shared resources such as cache, memory and network bandwidth towards<br>
obtaining Quality of Service. RDT will enable fine grain control of<br>
resources which in particular is valuable in cloud environments to meet<br>
Service Level Agreements while increasing resource utilization through<br>
sharing. CAT is a part of RDT and concerns itself with reserving for a<br>
process(es) a portion of last level cache with further fine grain<br>
control as to how much for code versus data. The below figure shows a<br>
single processor composed of 4 cores and the cache hierarchy. The L1<br>
cache is split into Instruction and Data, the L2 cache is next in speed<br>
to L1. The L1 and L2 caches are per core. The Last Level Cache (LLC) is<br>
shared among all cores. With CAT on the currently available hardware the<br>
LLC can be partitioned on a per process (virtual machine, container, or<br>
normal application) or process group basis.<br>
<br>
<br>
<br>
Libvirt and OpenStack [2] already support monitoring cache (CMT) and<br>
memory bandwidth usage local to a processor socket (MBM_local) and total<br>
memory bandwidth usage across all processor sockets (MBM_total) for a<br>
process or process group.<br>
<br>
<br>
<br>
<br></span>
2. How CAT works **<span class=""><br>
<br>
To learn more about CAT please refer to the Intel Processor Soft<br>
Developer's Manual<br></span>
<<a href="http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html" rel="noreferrer" target="_blank">http://www.intel.com/content/<wbr>www/us/en/processors/architect<wbr>ures-software-developer-<wbr>manuals.html</a>><span class=""><br>
volume 3b, chapters 17.16 and 17.17 [3]. Linux kernel support for the<br>
same is expected in release 4.10 and documented at [4]<br>
<br>
<br></span>
3. Libvirt Interface**<span class=""><br>
<br>
<br>
Libvirt support for CAT is underway with the patch at reversion 7<br>
<br>
<br>
<br>
Interface changes of libvirt:<br>
<br>
<br>
<br></span>
3.1 The capabilities xml has been extended to reveal cache information **<span class=""><br>
<br>
<br>
<br>
<cache><br>
<br>
<bank id='0' type='l3' size='56320' unit='KiB' cpus='0-21,44-65'><br>
<br>
<control min='2816' reserved='2816' unit='KiB' scope='L3'/><br>
<br>
</bank><br>
<br>
<bank id='1' type='l3' size='56320' unit='KiB' cpus='22-43,66-87'><br>
<br>
<control min='2816' reserved='2816' unit='KiB' scope='L3'/><br>
<br>
</bank><br>
<br>
</cache><br>
<br>
<br>
<br></span>
The new `cache` xml element shows that the host has two *banks* of<br>
*type* L3 or Last Level Cache (LLC), one per processor socket. The cache<br>
*type* is l3 cache, its *size* 56320 KiB, and the *cpus* attribute<span class=""><br>
indicates the physical CPUs associated with the same, here ‘0-21’,<br>
‘44-65’ respectively.<br>
<br>
<br>
<br></span>
The *control *tag shows that bank belongs to scope L3, with a minimum<span class=""><br>
possible allocation of 2816 KiB and still has 2816 KiB need to be reserved.<br>
<br>
<br>
<br>
If the host enabled CDP (Code and Data Prioritization) , l3 cache will<br>
be divided as code (L3CODE)and data (L3Data).<br>
<br>
<br>
<br>
Control tag will be extended to:<br>
<br>
...<br>
<br>
<control min='2816' reserved='2816' unit='KiB' scope='L3CODE'/><br>
<br>
<control min='2816' reserved='2816' unit='KiB' scope='L3DATA'/><br>
<br>
…<br>
<br>
<br>
<br>
The scope of L3CODE and L3DATA show that we can allocate cache for<br>
code/data usage respectively, they share same amount of l3 cache.<br>
<br>
<br>
<br></span>
3.2 Domain xml extended to include new CacheTune element **<span class=""><br>
<br>
<br>
<br>
<cputune><br>
<br>
<vcpupin vcpu='0' cpuset='0'/><br>
<br>
<vcpupin vcpu='1' cpuset='1'/><br>
<br>
<vcpupin vcpu='2' cpuset='0'/><br>
<br>
<vcpupin vcpu='3' cpuset='1'/><br>
<br>
<cachetune id='0' host_id='0' type='l3' size='2816' unit='KiB'<br>
vcpus='0, 1/><br>
<br>
<cachetune id='1' host_id='1' type='l3' size='2816' unit='KiB'<br>
vcpus=’2, 3’/><br>
<br>
...<br>
<br>
</cputune><br>
<br>
<br>
<br>
This means the guest will be have vcpus 0, 1 running on host’s socket 0,<br>
with 2816 KiB cache exclusively allocated to it and vcpus 2, 3 running<br>
on host’s socket 0, with 2816 KiB cache exclusively allocated to it.<br>
<br>
<br>
<br>
Here we need to make sure vcpus 0, 1 are pinned to the pcpus of socket<br>
0, refer capabilities<br>
<br>
<bank id='0' type='l3' size='56320' unit='KiB' cpus='0-21,44-65'>:<br>
<br>
<br>
<br>
Here we need to make sure vcpus 2, 3 are pinned to the pcpus of socket<br>
1, refer capabilities<br>
<br>
<bank id='1' type='l3' size='56320' unit='KiB' cpus='22-43,66-87'>:.<br>
<br>
<br>
<br></span>
3.3 Libvirt work flow for CAT**<br>
<br>
<br>
<br>
1. Create qemu process and get it’s PIDs<br>
2. Define a new resource control domain also known as<br>
*Cl*ass-*o*f-*S*ervice (CLOS) under /sys/fs/resctrl and set the<br>
desired *C*ache *B*it *M*ask(CBM) in the libvirt domain xml file in<span class=""><br>
addition to updating the default schemata of the host<br>
<br>
<br>
<br></span>
4. Proposed Nova Changes**<br>
<br>
<br>
<br>
1. Get host capabilities from libvirt and extend compute node’ filed<br>
2. Add new scheduler filter and weight to help schedule host for<br>
requested guest.<br>
3. Extend flavor’s (and image meta) extra spec fields:<span class=""><br>
<br>
<br>
<br>
We need to specify numa setting for NUMA hosts if we want to enable<br>
CAT, see [5] to learn more about NUMA.<br>
<br>
In flavor, we can have:<br>
<br>
<br>
<br>
vcpus=8<br>
<br>
mem=4<br>
<br>
hw:numa_nodes=2 - numa of NUMA nodes to expose to the guest.<br>
<br>
hw:numa_cpus.0=0,1,2,3,4,5<br>
<br>
hw:numa_cpus.1=6,7<br>
<br>
hw:numa_mem.0=3072<br>
<br>
hw:numa_mem.1=1024<br>
<br>
// new added in the proposal<br>
<br></span>
hw:cache_banks=2 ///cache banks to be allocated to a guest, (can be<br>
less than the number of NUMA nodes)/<br>
<br>
hw:cache_type.0=l3 ///cache bank type, could be l3, l3data + l3code/<br>
<br>
hw:cache_type.1=l3_c+d ///cache bank type, could be l3, l3data + l3code/<br>
<br>
hw:cache_vcpus.0=0,1 ///vcpu list on cache banks, can be none/<br>
<br>
hw:cache_vcpus.1=6,7<br>
<br>
hw:cache_l3.0=2816 ///cache size in KiB./<span class=""><br>
<br>
hw:cache_l3_code.1=2816<br>
<br>
hw:cache_l3_data.1=2816<br>
<br>
<br>
<br>
Here, user can clear about which vcpus will benefit cache allocation,<br>
about cache bank, it’s should be co-work with numa cell, it will<br>
allocate cache on a physical CPU socket, but here cache bank is a logic<br>
concept. Cache bank will allocate cache for a vcpu list, all vcpu list<br>
should group<br>
<br>
<br>
<br>
Modify in addition the <cachetune> element in libvirt domain xml, see<br>
3.2 for detail<br>
<br>
<br>
<br>
This will allocate 2 cache banks from the host’s cache banks and<br>
associate vcpus to the same.<br>
<br>
In the example, the guest will be have vcpus 0, 1 running on socket 0 of<br>
the host with 2816 KiB of cache for exclusive use and have vcpus 6, 7<br>
running on socket 1 of the host with l3 code cache 2816KiB and l3 data<br>
with 2816KiB cache allocation.<br>
<br>
<br>
<br>
If a NUMA Cell were to contain multiple CPU sockets (this is rare), then<br>
we will adjust NUMA vCPU placement policy, to ensure that vCPUs and the<br>
cache allocated to them are all co-located on the same socket.<br>
<br>
<br>
<br></span>
* We can define less cache bank on a multiple NUMA cell node.<br>
* No cache_vcpus parameter needs to be specified if no reservation is<span class=""><br>
desired.<br>
<br>
<br>
<br>
NOTE: the cache allocation for a guest is in isolated/exclusive mode.<br>
<br>
<br>
<br></span>
References**<span class=""><br>
<br>
<br>
<br>
[1]<br>
<a href="http://www.intel.com/content/www/us/en/architecture-and-technology/resource-director-technology.html" rel="noreferrer" target="_blank">http://www.intel.com/content/w<wbr>ww/us/en/architecture-and-tech<wbr>nology/resource-director-techn<wbr>ology.html</a><br>
<br>
[2] <a href="https://blueprints.launchpad.net/nova/+spec/support-perf-event" rel="noreferrer" target="_blank">https://blueprints.launchpad.n<wbr>et/nova/+spec/support-perf-eve<wbr>nt</a><br>
<br>
[3]<br>
<a href="http://www.intel.com/content/www/us/en/processors/architectures-software-developer-manuals.html" rel="noreferrer" target="_blank">http://www.intel.com/content/w<wbr>ww/us/en/processors/architectu<wbr>res-software-developer-<wbr>manuals.html</a><br>
<br>
[4]<br>
<a href="https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git/tree/Documentation/x86/intel_rdt_ui.txt?h=x86/cache" rel="noreferrer" target="_blank">https://git.kernel.org/cgit/li<wbr>nux/kernel/git/tip/tip.git/tre<wbr>e/Documentation/x86/intel_rdt_<wbr>ui.txt?h=x86/cache</a><br>
<br>
<br>
[5]<br>
<a href="https://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/virt-driver-numa-placement.html" rel="noreferrer" target="_blank">https://specs.openstack.org/op<wbr>enstack/nova-specs/specs/juno/<wbr>implemented/virt-driver-numa-<wbr>placement.html</a><br>
<br>
<br>
<br>
<br>
<br>
<br>
Best Regards<br>
<br>
<br>
<br>
Eli Qiao(乔立勇）OpenStack Core team OTC Intel.<br>
<br>
--<br>
<br>
<br>
<br>
<br>
<br></span><span class="">
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
</span></blockquote><div class="HOEnZb"><div class="h5">
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
</div></div></blockquote></div><br></div></div></div>