<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=koi8-r">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Dear Openstack user community,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have a compute node with 2 numa nodes and I would like to create 2 vms, each one using a different numa node through numa affinity with cpu, memory and nvme pci devices.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Openstack flavor<o:p></o:p></p>
<p class="MsoNormal">openstack flavor create --public xlarge.numa.perf.test --ram 200000 --disk 700 --vcpus 20 --property hw:cpu_policy=dedicated --property hw:emulator_threads_policy=isolate --property hw:numa_nodes='1' --property pci_passthrough:alias='nvme:4'<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The first vm is successfully created<o:p></o:p></p>
<p class="MsoNormal">openstack server create --network hpc --flavor xlarge.numa.perf.test --image centos7.6-image --availability-zone nova:zeus-53.localdomain --key-name mykey kudu-1<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">However the second vm fails<o:p></o:p></p>
<p class="MsoNormal">openstack server create --network hpc --flavor xlarge.numa.perf --image centos7.6-kudu-image --availability-zone nova:zeus-53.localdomain --key-name mykey kudu-4<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Errors in nova compute node<o:p></o:p></p>
<p class="MsoNormal">2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [req-b5a25c73-8c7d-466c-8128-71f29e7ae8aa 91e83343e9834c8ba0172ff369c8acac b91520cff5bd45c59a8de07c38641582 - default default] [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] Instance
failed to spawn: libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-09-27T06:45:19.118089Z qemu-kvm: kvm_init_vcpu failed: Cannot allocate memory<o:p></o:p></p>
<p class="MsoNormal">2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae] libvirtError: internal error: qemu unexpectedly closed the monitor: 2019-09-27T06:45:19.118089Z qemu-kvm: kvm_init_vcpu failed: Cannot
allocate memory<o:p></o:p></p>
<p class="MsoNormal">2019-09-27 16:45:19.785 7 ERROR nova.compute.manager [instance: ebe4e78c-501e-4535-ae15-948301cbf1ae]<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Numa cell/node 1 (the one assigned on kudu-4) has enough cpu, memory, pci devices and disk capacity to fit this vm. NOTE: below is the information relevant I could think of that shows resources available after creating the second vm.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@zeus-53 ~]# numactl -H<o:p></o:p></p>
<p class="MsoNormal">available: 2 nodes (0-1)<o:p></o:p></p>
<p class="MsoNormal">node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 28 29 30 31 32 33 34 35 36 37 38 39 40 41<o:p></o:p></p>
<p class="MsoNormal">node 0 size: 262029 MB<o:p></o:p></p>
<p class="MsoNormal">node 0 free: 52787 MB<o:p></o:p></p>
<p class="MsoNormal">node 1 cpus: 14 15 16 17 18 19 20 21 22 23 24 25 26 27 42 43 44 45 46 47 48 49 50 51 52 53 54 55<o:p></o:p></p>
<p class="MsoNormal">node 1 size: 262144 MB<o:p></o:p></p>
<p class="MsoNormal">node 1 free: 250624 MB<o:p></o:p></p>
<p class="MsoNormal">node distances:<o:p></o:p></p>
<p class="MsoNormal">node 0 1<o:p></o:p></p>
<p class="MsoNormal"> 0: 10 21<o:p></o:p></p>
<p class="MsoNormal"> 1: 21 10<o:p></o:p></p>
<p class="MsoNormal">NOTE: this is to show that numa node/cell 1 has enough resources available (also nova-compute logs shows that kudu-4 is assigned to cell 1)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@zeus-53 ~]# df -h<o:p></o:p></p>
<p class="MsoNormal">Filesystem Size Used Avail Use% Mounted on<o:p></o:p></p>
<p class="MsoNormal">/dev/md127 3.7T 9.1G 3.7T 1% /<o:p></o:p></p>
<p class="MsoNormal">...<o:p></o:p></p>
<p class="MsoNormal">NOTE: vm disk files goes to root (/) partition<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@zeus-53 ~]# lsblk<o:p></o:p></p>
<p class="MsoNormal">NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT<o:p></o:p></p>
<p class="MsoNormal">sda 8:0 0 59.6G 0 disk<o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Arial","sans-serif"">†</span>€sda1 8:1 0 1G 0 part /boot<o:p></o:p></p>
<p class="MsoNormal">„€sda2 8:2 0 16G 0 part [SWAP]<o:p></o:p></p>
<p class="MsoNormal">nvme0n1 259:8 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">„€nvme0n1p1 259:9 0 1.8T 0 part<o:p></o:p></p>
<p class="MsoNormal"> „€md127 9:127 0 3.7T 0 raid0 /<o:p></o:p></p>
<p class="MsoNormal">nvme1n1 259:6 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">„€nvme1n1p1 259:7 0 1.8T 0 part<o:p></o:p></p>
<p class="MsoNormal"> „€md127 9:127 0 3.7T 0 raid0 /<o:p></o:p></p>
<p class="MsoNormal">nvme2n1 259:2 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">nvme3n1 259:1 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">nvme4n1 259:0 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">nvme5n1 259:3 0 1.8T 0 disk<o:p></o:p></p>
<p class="MsoNormal">NOTE: this is to show that there are 4 nvme disks (nvme2n1, nvme3n1, nvme4n1, nvme5n1) available for the second vm<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What "emu-kvm: kvm_init_vcpu failed: Cannot allocate memory" means in this context?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you very much<span style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#4D00C7">
</span><span lang="EN-AU" style="font-size:10.0pt;font-family:"Arial","sans-serif";color:#4D00C7"><o:p></o:p></span></p>
</div>
<div style="font-size:8pt; font-family: Arial">NOTICE</div>
<div style="font-size:8pt; font-family: Arial">Please consider the environment before printing this email. This message and any attachments are intended for the addressee named and may contain legally privileged/confidential/copyright information. If you are
not the intended recipient, you should not read, use, disclose, copy or distribute this communication. If you have received this message in error please notify us at once by return email and then delete both messages. We accept no liability for the distribution
of viruses or similar in electronic communications. This notice should not be removed.
</div>
</body>
</html>