[wallaby][nova] CPU topology and NUMA Nodes
Ammad Syed
syedammad83 at gmail.com
Tue Jun 29 16:35:01 UTC 2021
Hi Stephen,
I have checked all cpus are online.
root at kvm10-a1-khi01:/etc/nova# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 48 bits physical, 48 bits virtual
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 4
Vendor ID: AuthenticAMD
CPU family: 21
I have made below configuration in nova.conf.
[compute]
cpu_shared_set = 2-7,10-15,18-23,26-31
Below is the xml in nova logs that nova is trying to create domain.
2021-06-29 16:30:56.576 2819 ERROR nova.virt.libvirt.guest
[req-c76c6809-1775-43a8-bfb1-70f6726cad9d 2af528fdf3244e15b4f3f8fcfc0889c5
890eb2b7d1b8488aa88de7c34d08817a - default default] Error launching a
defined domain with XML: <domain type='kvm'>
<name>instance-0000026d</name>
<uuid>06ff4fd5-b21f-4f64-9dde-55e86dd15da6</uuid>
<metadata>
<nova:instance xmlns:nova="http://openstack.org/xmlns/libvirt/nova/1.1">
<nova:package version="23.0.0"/>
<nova:name>cpu</nova:name>
<nova:creationTime>2021-06-29 16:30:50</nova:creationTime>
<nova:flavor name="RC-16G">
<nova:memory>16384</nova:memory>
<nova:disk>5</nova:disk>
<nova:swap>0</nova:swap>
<nova:ephemeral>0</nova:ephemeral>
<nova:vcpus>8</nova:vcpus>
</nova:flavor>
<nova:owner>
<nova:user uuid="2af528fdf3244e15b4f3f8fcfc0889c5">admin</nova:user>
<nova:project
uuid="890eb2b7d1b8488aa88de7c34d08817a">admin</nova:project>
</nova:owner>
<nova:root type="image" uuid="1024317b-0db6-418c-bc39-de9b61d8ce59"/>
<nova:ports>
<nova:port uuid="dccf68a2-ec48-4985-94ce-b3487cfc99f3">
<nova:ip type="fixed" address="192.168.100.106" ipVersion="4"/>
</nova:port>
</nova:ports>
</nova:instance>
</metadata>
<memory unit='KiB'>16777216</memory>
<currentMemory unit='KiB'>16777216</currentMemory>
<vcpu placement='static' cpuset='2-7,10-15,18-23,26-31'>8</vcpu>
<cputune>
<shares>8192</shares>
</cputune>
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>OpenStack Foundation</entry>
<entry name='product'>OpenStack Nova</entry>
<entry name='version'>23.0.0</entry>
<entry name='serial'>06ff4fd5-b21f-4f64-9dde-55e86dd15da6</entry>
<entry name='uuid'>06ff4fd5-b21f-4f64-9dde-55e86dd15da6</entry>
<entry name='family'>Virtual Machine</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
<boot dev='hd'/>
<smbios mode='sysinfo'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='custom' match='exact' check='partial'>
<model fallback='allow'>Opteron_G5</model>
<topology sockets='4' cores='2' threads='1'/>
</cpu>
<clock offset='utc'>
<timer name='pit' tickpolicy='delay'/>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/>
<source
file='/var/lib/nova/instances/06ff4fd5-b21f-4f64-9dde-55e86dd15da6/disk'/>
<target dev='sda' bus='scsi'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='scsi' index='0' model='virtio-scsi'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</controller>
<controller type='usb' index='0' model='piix3-uhci'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<interface type='bridge'>
<mac address='fa:16:3e:f5:46:7d'/>
<source bridge='br-int'/>
<virtualport type='openvswitch'>
<parameters interfaceid='dccf68a2-ec48-4985-94ce-b3487cfc99f3'/>
</virtualport>
<target dev='tapdccf68a2-ec'/>
<model type='virtio'/>
<driver name='vhost' queues='8'/>
<mtu size='1442'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
<serial type='pty'>
<log
file='/var/lib/nova/instances/06ff4fd5-b21f-4f64-9dde-55e86dd15da6/console.log'
append='off'/>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<log
file='/var/lib/nova/instances/06ff4fd5-b21f-4f64-9dde-55e86dd15da6/console.log'
append='off'/>
<target type='serial' port='0'/>
</console>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='spice' autoport='yes' listen='0.0.0.0'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'
primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<stats period='10'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</memballoon>
<rng model='virtio'>
<backend model='random'>/dev/urandom</backend>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06'
function='0x0'/>
</rng>
</devices>
</domain>
: libvirt.libvirtError: Unable to write to
'/sys/fs/cgroup/cpuset/machine.slice/machine-qemu\x2d1\x2dinstance\x2d0000026d.scope/emulator/cpuset.cpus':
Permission denied
Ammad
On Tue, Jun 29, 2021 at 8:29 PM Stephen Finucane <stephenfin at redhat.com>
wrote:
> On Tue, 2021-06-29 at 17:44 +0500, Ammad Syed wrote:
> > Thanks,, the information is really helpful. I am have set below
> properties to
> > flavor according to my numa policies.
> >
> > --property hw:numa_nodes=FLAVOR-NODES \
> > --property hw:numa_cpus.N=FLAVOR-CORES \
> > --property hw:numa_mem.N=FLAVOR-MEMORY
> >
> > I am having below error in compute logs. Any advise.
> >
> > libvirt.libvirtError: Unable to write to
> > '/sys/fs/cgroup/cpuset/machine.slice/machine-
> > qemu\x2d48\x2dinstance\x2d0000026b.scope/emulator/cpuset.cpus':
> Permission
> > denied
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest Traceback
> (most
> > recent call last):
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest File
> > "/usr/lib/python3/dist-packages/nova/virt/libvirt/guest.py", line 155,
> in launch
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest return
> > self._domain.createWithFlags(flags)
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest File
> > "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 193, in doit
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest result
> =
> > proxy_call(self._autowrap, f, *args, **kwargs)
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest File
> > "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 151, in
> proxy_call
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest rv =
> > execute(f, *args, **kwargs)
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest File
> > "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 132, in execute
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest
> six.reraise(c,
> > e, tb)
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest File
> > "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest raise
> value
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest File
> > "/usr/lib/python3/dist-packages/eventlet/tpool.py", line 86, in tworker
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest rv =
> > meth(*args, **kwargs)
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest File
> > "/usr/lib/python3/dist-packages/libvirt.py", line 1265, in
> createWithFlags
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest if ret
> == -1:
> > raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest
> > libvirt.libvirtError: Unable to write to
> > '/sys/fs/cgroup/cpuset/machine.slice/machine-
> > qemu\x2d48\x2dinstance\x2d0000026b.scope/emulator/cpuset.cpus':
> Permission
> > denied
> > 2021-06-29 12:33:10.144 1310945 ERROR nova.virt.libvirt.guest
> > 2021-06-29 12:33:10.146 1310945 ERROR nova.virt.libvirt.driver
> [req-4f6fc6aa-
> > 04d6-4dc0-921f-2913b40a76a9 2af528fdf3244e15b4f3f8fcfc0889c5
> > 890eb2b7d1b8488aa88de7c34d08817a - default default] [instance:
> ed87bf68-b631-
> > 4a00-9eb5-22d32ec37402] Failed to start libvirt guest:
> libvirt.libvirtError:
> > Unable to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-
> > qemu\x2d48\x2dinstance\x2d0000026b.scope/emulator/cpuset.cpus':
> Permission
> > denied
> > 2021-06-29 12:33:10.150 1310945 INFO os_vif [req-4f6fc6aa-04d6-4dc0-921f-
> > 2913b40a76a9 2af528fdf3244e15b4f3f8fcfc0889c5
> 890eb2b7d1b8488aa88de7c34d08817a -
> > default default] Successfully unplugged vif
> > VIFOpenVSwitch(active=False,address=fa:16:3e:ba:3d:c8,bridge_name='br-
> > int',has_traffic_filtering=True,id=a991cd33-2610-4823-a471-
> > 62171037e1b5,network=Network(a0d85af2-a991-4102-8453-
> >
> ba68c5e10b65),plugin='ovs',port_profile=VIFPortProfileOpenVSwitch,preserve_on_de
> > lete=False,vif_name='tapa991cd33-26')
> > 2021-06-29 12:33:10.151 1310945 INFO nova.virt.libvirt.driver
> [req-4f6fc6aa-
> > 04d6-4dc0-921f-2913b40a76a9 2af528fdf3244e15b4f3f8fcfc0889c5
> > 890eb2b7d1b8488aa88de7c34d08817a - default default] [instance:
> ed87bf68-b631-
> > 4a00-9eb5-22d32ec37402] Deleting instance files
> > /var/lib/nova/instances/ed87bf68-b631-4a00-9eb5-22d32ec37402_del
> > 2021-06-29 12:33:10.152 1310945 INFO nova.virt.libvirt.driver
> [req-4f6fc6aa-
> > 04d6-4dc0-921f-2913b40a76a9 2af528fdf3244e15b4f3f8fcfc0889c5
> > 890eb2b7d1b8488aa88de7c34d08817a - default default] [instance:
> ed87bf68-b631-
> > 4a00-9eb5-22d32ec37402] Deletion of
> /var/lib/nova/instances/ed87bf68-b631-4a00-
> > 9eb5-22d32ec37402_del complete
> > 2021-06-29 12:33:10.258 1310945 ERROR nova.compute.manager
> [req-4f6fc6aa-04d6-
> > 4dc0-921f-2913b40a76a9 2af528fdf3244e15b4f3f8fcfc0889c5
> > 890eb2b7d1b8488aa88de7c34d08817a - default default] [instance:
> ed87bf68-b631-
> > 4a00-9eb5-22d32ec37402] Instance failed to spawn: libvirt.libvirtError:
> Unable
> > to write to '/sys/fs/cgroup/cpuset/machine.slice/machine-
> > qemu\x2d48\x2dinstance\x2d0000026b.scope/emulator/cpuset.cpus':
> Permission
> > denied
> >
> > Any advise how to fix this permission issue ?
> >
> > I have manually created the directory machine-qemu in
> > /sys/fs/cgroup/cpuset/machine.slice/ but still having the same error.
> >
> > I have also tried to set [compute] cpu_shared_set AND [compute]
> > cpu_dedicated_set they are also giving the same error.
>
> There are quite a few bugs about this [1][2]. It seems most of them are
> caused
> by CPUs being offlined. Have you offline CPUs? Are the CPUs listed in the
> mask
> all available?
>
> Stephen
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1609785
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1842716
>
> > Using ubuntu20.04 and qemu-kvm 4.2.
> >
> > Ammad
> >
> > On Fri, Jun 25, 2021 at 10:54 AM Sean Mooney <smooney at redhat.com> wrote:
> > > On Fri, 2021-06-25 at 10:02 +0500, Ammad Syed wrote:
> > > > Hi,
> > > >
> > > > I am using openstack wallaby on ubuntu 20.04 and kvm. I am working
> to make
> > > > optimized flavor properties that should provide optimal performance.
> I was
> > > > reviewing the document below.
> > > >
> > > > https://docs.openstack.org/nova/wallaby/admin/cpu-topologies.html
> > > >
> > > > I have two socket AMD compute node. The workload running on nodes
> are mixed
> > > > workload.
> > > >
> > > > My question is should I use default nova CPU topology and NUMA node
> that
> > > > nova deploys instance by default OR should I use hw:cpu_sockets='2'
> > > > and hw:numa_nodes='2'.
> > > the latter hw:cpu_sockets='2' and hw:numa_nodes='2' should give you
> better
> > > performce
> > > however you should also set hw:mem_page_size=small or
> hw:mem_page_size=any
> > > when you enable virtual numa policies we afinities the guest memory to
> host
> > > numa nodes.
> > > This can lead to Out of memory evnet on the the host numa nodes which
> can
> > > result in vms
> > > being killed by the host kernel memeory reaper if you do not enable
> numa aware
> > > memeory
> > > trackign iin nova which is done by setting hw:mem_page_size. setting
> > > hw:mem_page_size has
> > > the side effect of of disabling memory over commit so you have to bare
> that in
> > > mind.
> > > if you are using numa toplogy you should almost always also use
> hugepages
> > > which are enabled
> > > using hw:mem_page_size=large this however requires you to configure
> > > hupgepages in the host
> > > at boot.
> > > >
> > > > Which one from above provide best instance performance ? or any other
> > > > tuning should I do ?
> > >
> > > in the libvirt driver the default cpu toplogy we will genergated
> > > is 1 thread per core, 1 core per socket and 1 socket per flavor.vcpu.
> > > (technially this is an undocumeted implemation detail that you should
> not rely
> > > on, we have the hw:cpu_* element if you care about the toplogy)
> > >
> > > this was more effincet in the early days of qemu/openstack but has may
> issue
> > > when software is chagne per sokcet or oepreating systems have
> > > a limit on socket supported such as windows.
> > >
> > > generally i advies that you set hw:cpu_sockets to the typical number of
> > > sockets on the underlying host.
> > > simialrly if the flavor will only be run on host with SMT/hypertreading
> > > enabled on you shoudl set hw:cpu_threads=2
> > >
> > > the flavor.vcpus must be devisable by the product of hw:cpu_sockets,
> > > hw:cpu_cores and hw:cpu_threads if they are set.
> > >
> > > so if you have hw:cpu_threads=2 it must be devisable by 2
> > > if you have hw:cpu_threads=2 and hw:cpu_sockets=2 flavor.vcpus must
> be a
> > > multiple of 4
> > > >
> > > > The note in the URL (CPU topology sesion) suggests that I should
> stay with
> > > > default options that nova provides.
> > > in generaly no you should aling it to the host toplogy if you have
> similar
> > > toplogy across your data center.
> > > the default should always just work but its not nessisarly optimal and
> window
> > > sguest might not boot if you have too many sockets.
> > > windows 10 for exmple only supprot 2 socket so you could only have 2
> > > flavor.vcpus if you used the default toplogy.
> > >
> > > >
> > > > Currently it also works with libvirt/QEMU driver but we don’t
> recommend it
> > > > in production use cases. This is because vCPUs are actually running
> in one
> > > > thread on host in qemu TCG (Tiny Code Generator), which is the
> backend for
> > > > libvirt/QEMU driver. Work to enable full multi-threading support for
> TCG
> > > > (a.k.a. MTTCG) is on going in QEMU community. Please see this MTTCG
> project
> > > > <http://wiki.qemu.org/Features/tcg-multithread> page for detail.
> > > we do not gnerally recommende using qemu without kvm in produciton.
> > > the mttcg backend is useful in cases where you want to emulate other
> plathform
> > > but that usecsae
> > > is not currently supported in nova.
> > > for your deployment you should use libvirt with kvm and you should also
> > > consider if you want to support
> > > nested virtualisation or not.
> > > >
> > > >
> > > > Ammad
> > >
> > >
> >
> >
>
>
>
--
Regards,
Syed Ammad Ali
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20210629/ed00a97f/attachment-0001.html>
More information about the openstack-discuss
mailing list