Issue with launching instance with OVS-DPDK
Hello I've built an Openstack all-in-one using OVS-DPDK via Devstack. I can launch instances which use the "m1.small" flavour (which I have modified to include the hw:mem_size large as per the DPDK instructions) but as soon as I try to launch anything more than m1.small, I get this error: Jan 28 12:56:52 localhost nova-conductor: #033[01;31mERROR nova.scheduler.utils [#033[01;36mNone req-917cd3b9-8ce6-41af-8d44-045002512c91 #033[00;36madmin admin#033[01;31m] #033[01;35m[instance: 25cfee28-08e9-419c-afdb-4d0fe515fb2a] #033[01;31mError from last host: localhost (node localhost): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1935, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2215, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 25cfee28-08e9-419c-afdb-4d0fe515fb2a was re-scheduled: internal error: qemu unexpectedly closed the monitor: 2019-01-28T12:56:48.127594Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu46b3c508-f8,server: info: QEMU waiting for connection on: disconnected:unix:/var/run/openvswitch/vhu46b3c508-f8,server\n2019-01-28T12:56:49.251071Z qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/4-instance-00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m My Hypervisor is reporting 510.7GB of RAM and 61 vCPUs. Build is the latest git clone of Devstack. Thanks David
On Mon, 2019-01-28 at 13:17 +0000, David Lake wrote:
Hello
I’ve built an Openstack all-in-one using OVS-DPDK via Devstack.
I can launch instances which use the “m1.small” flavour (which I have modified to include the hw:mem_size large as per the DPDK instructions) but as soon as I try to launch anything more than m1.small, I get this error:
Jan 28 12:56:52 localhost nova-conductor: #033[01;31mERROR nova.scheduler.utils [#033[01;36mNone req-917cd3b9-8ce6- 41af-8d44-045002512c91 #033[00;36madmin admin#033[01;31m] #033[01;35m[instance: 25cfee28-08e9-419c-afdb-4d0fe515fb2a] #033[01;31mError from last host: localhost (node localhost): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1935, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2215, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 25cfee28-08e9- 419c-afdb-4d0fe515fb2a was re-scheduled: internal error: qemu unexpectedly closed the monitor: 2019-01- 28T12:56:48.127594Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu46b3c508-f8,server: info: QEMU waiting for connection on: disconnected:unix:/var/run/openvswitch/vhu46b3c508-f8,server\n2019-01-28T12:56:49.251071Z qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/libvirt/qemu/4-instance- 00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m
My Hypervisor is reporting 510.7GB of RAM and 61 vCPUs. how much of that ram did you allocate as hugepages.
can you provide the output of cat /proc/meminfo
Build is the latest git clone of Devstack.
Thanks
David
Answers <DL> in-line </DL> Thanks David -----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: 29 January 2019 14:55 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack-dev@lists.openstack.org Cc: Ge, Chang Dr (Elec Electronic Eng) <c.ge@surrey.ac.uk> Subject: Re: Issue with launching instance with OVS-DPDK On Mon, 2019-01-28 at 13:17 +0000, David Lake wrote:
Hello
I’ve built an Openstack all-in-one using OVS-DPDK via Devstack.
I can launch instances which use the “m1.small” flavour (which I have modified to include the hw:mem_size large as per the DPDK instructions) but as soon as I try to launch anything more than m1.small, I get this error:
Jan 28 12:56:52 localhost nova-conductor: #033[01;31mERROR nova.scheduler.utils [#033[01;36mNone req-917cd3b9-8ce6- 41af-8d44-045002512c91 #033[00;36madmin admin#033[01;31m] #033[01;35m[instance: 25cfee28-08e9-419c-afdb-4d0fe515fb2a] #033[01;31mError from last host: localhost (node localhost): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1935, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2215, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 25cfee28-08e9- 419c-afdb-4d0fe515fb2a was re-scheduled: internal error: qemu unexpectedly closed the monitor: 2019-01- 28T12:56:48.127594Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu46b3c508-f8,server: info: QEMU waiting for connection on: disconnected:unix:/var/run/openvswitch/vhu46b3c508-f8,server\n2019-01- 28T12:56:49.251071Z qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/ libvirt/qemu/4-instance- 00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m
My Hypervisor is reporting 510.7GB of RAM and 61 vCPUs. how much of that ram did you allocate as hugepages.
<DL> OVS_NUM_HUGEPAGES=3072 </DL> can you provide the output of cat /proc/meminfo <DL> MemTotal: 526779552 kB MemFree: 466555316 kB MemAvailable: 487218548 kB Buffers: 2308 kB Cached: 22962972 kB SwapCached: 0 kB Active: 29493384 kB Inactive: 13344640 kB Active(anon): 20826364 kB Inactive(anon): 522012 kB Active(file): 8667020 kB Inactive(file): 12822628 kB Unevictable: 43636 kB Mlocked: 47732 kB SwapTotal: 4194300 kB SwapFree: 4194300 kB Dirty: 20 kB Writeback: 0 kB AnonPages: 19933028 kB Mapped: 171680 kB Shmem: 1450564 kB Slab: 1224444 kB SReclaimable: 827696 kB SUnreclaim: 396748 kB KernelStack: 69392 kB PageTables: 181020 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 261292620 kB Committed_AS: 84420252 kB VmallocTotal: 34359738367 kB VmallocUsed: 1352128 kB VmallocChunk: 34154915836 kB HardwareCorrupted: 0 kB AnonHugePages: 5365760 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 6144 HugePages_Free: 2048 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 746304 kB DirectMap2M: 34580480 kB DirectMap1G: 502267904 kB [stack@localhost devstack]$ </DL>
Build is the latest git clone of Devstack.
Thanks
David
Answers <DL> in-line </DL>
Thanks
David
-----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: 29 January 2019 14:55 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack-dev@lists.openstack.org Cc: Ge, Chang Dr (Elec Electronic Eng) <c.ge@surrey.ac.uk> Subject: Re: Issue with launching instance with OVS-DPDK
On Mon, 2019-01-28 at 13:17 +0000, David Lake wrote:
Hello
I’ve built an Openstack all-in-one using OVS-DPDK via Devstack.
I can launch instances which use the “m1.small” flavour (which I have modified to include the hw:mem_size large as per the DPDK instructions) but as soon as I try to launch anything more than m1.small, I get this error:
Jan 28 12:56:52 localhost nova-conductor: #033[01;31mERROR nova.scheduler.utils [#033[01;36mNone req-917cd3b9-8ce6- 41af-8d44-045002512c91 #033[00;36madmin admin#033[01;31m] #033[01;35m[instance: 25cfee28-08e9-419c-afdb-4d0fe515fb2a] #033[01;31mError from last host: localhost (node localhost): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1935, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2215, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 25cfee28-08e9- 419c-afdb-4d0fe515fb2a was re-scheduled: internal error: qemu unexpectedly closed the monitor: 2019-01- 28T12:56:48.127594Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu46b3c508-f8,server: info: QEMU waiting for connection on: disconnected:unix:/var/run/openvswitch/vhu46b3c508-f8,server\n2019-01- 28T12:56:49.251071Z qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/ libvirt/qemu/4-instance- 00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m
My Hypervisor is reporting 510.7GB of RAM and 61 vCPUs.
how much of that ram did you allocate as hugepages.
<DL> OVS_NUM_HUGEPAGES=3072 </DL> ok so you used networking-ovs-dpdks ablitiy to automatically allocate 2MB hugepages at runtime so this should have allocate 6GB of hugepages per numa node.
can you provide the output of cat /proc/meminfo
<DL>
MemTotal: 526779552 kB MemFree: 466555316 kB MemAvailable: 487218548 kB Buffers: 2308 kB Cached: 22962972 kB SwapCached: 0 kB Active: 29493384 kB Inactive: 13344640 kB Active(anon): 20826364 kB Inactive(anon): 522012 kB Active(file): 8667020 kB Inactive(file): 12822628 kB Unevictable: 43636 kB Mlocked: 47732 kB SwapTotal: 4194300 kB SwapFree: 4194300 kB Dirty: 20 kB Writeback: 0 kB AnonPages: 19933028 kB Mapped: 171680 kB Shmem: 1450564 kB Slab: 1224444 kB SReclaimable: 827696 kB SUnreclaim: 396748 kB KernelStack: 69392 kB PageTables: 181020 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 261292620 kB Committed_AS: 84420252 kB VmallocTotal: 34359738367 kB VmallocUsed: 1352128 kB VmallocChunk: 34154915836 kB HardwareCorrupted: 0 kB AnonHugePages: 5365760 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 6144 since we have 6144 total and OVS_NUM_HUGEPAGES was set to 3072
On Tue, 2019-01-29 at 18:05 +0000, David Lake wrote: this indicate the host has 2 numa nodes
HugePages_Free: 2048 and you currently have 4G of 2MB hugepages free. however this will also be split across numa nodes.
the qemu commandline you provied which i have coppied below is trying to allocate 4G of hugepage memory from a single host numa node qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/ libvirt/qemu/4-instance- 00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m as a result the vm is failing to boot because nova cannot create the vm with a singel numa node. if you set hw:numa_nodes=2 this vm would likely boot but since you have a 512G hostyou should be able to increase OVS_NUM_HUGEPAGES to something like OVS_NUM_HUGEPAGES=14336. this will allocate 60G of 2MB hugepages total. if you want to allocate more then about 96G of hugepages you should set OVS_ALLOCATE_HUGEPAGES=False and instead allcoate the hugepages on the kernel commandline using 1G hugepages. e.g. default_hugepagesz=1G hugepagesz=1G hugepages=480 This is becase it take a long time for ovs-dpdk to scan all the hugepages on start up. setting default_hugepagesz=1G hugepagesz=1G hugepages=480 will leave 32G of ram for the host. if it a comptue node and not a contorller you can safly reduce the the free host ram to 16G e.g. default_hugepagesz=1G hugepagesz=1G hugepages=496 i would not advice allocating much more above than 496G of hugepages as the qemu emularot over head can eaially get into the 10s of gigs if you have 50+ vms running.
HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 746304 kB DirectMap2M: 34580480 kB DirectMap1G: 502267904 kB [stack@localhost devstack]$
</DL>
Build is the latest git clone of Devstack.
Thanks
David
Hi Sean I've set OVS_NUM_HUGEPAGES=14336 but now Devstack is failing to install... David full create: /opt/stack/tempest/.tox/tempest ERROR: invocation failed (exit code 1), logfile: /opt/stack/tempest/.tox/tempest/log/full-0.log ERROR: actionid: full msg: getenv cmdargs: '/usr/bin/python -m virtualenv --python /usr/bin/python tempest' Already using interpreter /usr/bin/python New python executable in /opt/stack/tempest/.tox/tempest/bin/python Complete output from command /opt/stack/tempest/.tox/tempest/bin/python -m pip config list: ERROR: unknown command "config" ---------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/virtualenv.py", line 2502, in <module> main() File "/usr/lib/python2.7/site-packages/virtualenv.py", line 793, in main symlink=options.symlink, File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1087, in create_environment install_wheel(to_install, py_executable, search_dirs, download=download) File "/usr/lib/python2.7/site-packages/virtualenv.py", line 935, in install_wheel _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs) File "/usr/lib/python2.7/site-packages/virtualenv.py", line 964, in _install_wheel_with_search_dir config = _pip_config(py_executable, python_path) File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1038, in _pip_config remove_from_env=["PIP_VERBOSE", "PIP_QUIET"], File "/usr/lib/python2.7/site-packages/virtualenv.py", line 886, in call_subprocess raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode)) OSError: Command /opt/stack/tempest/.tox/tempest/bin/python -m pip config list failed with error code 1 ERROR: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python -m virtualenv --python /usr/bin/python tempest (see /opt/stack/tempest/.tox/tempest/log/full-0.log)', 1) ___________________________________ summary ____________________________________ ERROR: full: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python -m virtualenv --python /usr/bin/python tempest (see /opt/stack/tempest/.tox/tempest/log/full-0.log)', 1) -----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: 29 January 2019 21:46 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack-dev@lists.openstack.org Cc: Ge, Chang Dr (Elec Electronic Eng) <c.ge@surrey.ac.uk> Subject: Re: Issue with launching instance with OVS-DPDK On Tue, 2019-01-29 at 18:05 +0000, David Lake wrote:
Answers <DL> in-line </DL>
Thanks
David
-----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: 29 January 2019 14:55 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack-dev@lists.openstack.org Cc: Ge, Chang Dr (Elec Electronic Eng) <c.ge@surrey.ac.uk> Subject: Re: Issue with launching instance with OVS-DPDK
On Mon, 2019-01-28 at 13:17 +0000, David Lake wrote:
Hello
I’ve built an Openstack all-in-one using OVS-DPDK via Devstack.
I can launch instances which use the “m1.small” flavour (which I have modified to include the hw:mem_size large as per the DPDK instructions) but as soon as I try to launch anything more than m1.small, I get this error:
Jan 28 12:56:52 localhost nova-conductor: #033[01;31mERROR nova.scheduler.utils [#033[01;36mNone req-917cd3b9-8ce6- 41af-8d44-045002512c91 #033[00;36madmin admin#033[01;31m] #033[01;35m[instance: 25cfee28-08e9-419c-afdb-4d0fe515fb2a] #033[01;31mError from last host: localhost (node localhost): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1935, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2215, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 25cfee28-08e9- 419c-afdb-4d0fe515fb2a was re-scheduled: internal error: qemu unexpectedly closed the monitor: 2019-01- 28T12:56:48.127594Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu46b3c508-f8,server: info: QEMU waiting for connection on: disconnected:unix:/var/run/openvswitch/vhu46b3c508-f8,server\n2019-0 1- 28T12:56:49.251071Z qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepage s/ libvirt/qemu/4-instance- 00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m
My Hypervisor is reporting 510.7GB of RAM and 61 vCPUs.
how much of that ram did you allocate as hugepages.
<DL> OVS_NUM_HUGEPAGES=3072 </DL> ok so you used networking-ovs-dpdks ablitiy to automatically allocate 2MB hugepages at runtime so this should have allocate 6GB of hugepages per numa node.
can you provide the output of cat /proc/meminfo
<DL>
MemTotal: 526779552 kB MemFree: 466555316 kB MemAvailable: 487218548 kB Buffers: 2308 kB Cached: 22962972 kB SwapCached: 0 kB Active: 29493384 kB Inactive: 13344640 kB Active(anon): 20826364 kB Inactive(anon): 522012 kB Active(file): 8667020 kB Inactive(file): 12822628 kB Unevictable: 43636 kB Mlocked: 47732 kB SwapTotal: 4194300 kB SwapFree: 4194300 kB Dirty: 20 kB Writeback: 0 kB AnonPages: 19933028 kB Mapped: 171680 kB Shmem: 1450564 kB Slab: 1224444 kB SReclaimable: 827696 kB SUnreclaim: 396748 kB KernelStack: 69392 kB PageTables: 181020 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 261292620 kB Committed_AS: 84420252 kB VmallocTotal: 34359738367 kB VmallocUsed: 1352128 kB VmallocChunk: 34154915836 kB HardwareCorrupted: 0 kB AnonHugePages: 5365760 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 6144 since we have 6144 total and OVS_NUM_HUGEPAGES was set to 3072 this indicate the host has 2 numa nodes HugePages_Free: 2048 and you currently have 4G of 2MB hugepages free. however this will also be split across numa nodes.
the qemu commandline you provied which i have coppied below is trying to allocate 4G of hugepage memory from a single host numa node qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/ libvirt/qemu/4-instance- 00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m as a result the vm is failing to boot because nova cannot create the vm with a singel numa node. if you set hw:numa_nodes=2 this vm would likely boot but since you have a 512G hostyou should be able to increase OVS_NUM_HUGEPAGES to something like OVS_NUM_HUGEPAGES=14336. this will allocate 60G of 2MB hugepages total. if you want to allocate more then about 96G of hugepages you should set OVS_ALLOCATE_HUGEPAGES=False and instead allcoate the hugepages on the kernel commandline using 1G hugepages. e.g. default_hugepagesz=1G hugepagesz=1G hugepages=480 This is becase it take a long time for ovs-dpdk to scan all the hugepages on start up. setting default_hugepagesz=1G hugepagesz=1G hugepages=480 will leave 32G of ram for the host. if it a comptue node and not a contorller you can safly reduce the the free host ram to 16G e.g. default_hugepagesz=1G hugepagesz=1G hugepages=496 i would not advice allocating much more above than 496G of hugepages as the qemu emularot over head can eaially get into the 10s of gigs if you have 50+ vms running.
HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 746304 kB DirectMap2M: 34580480 kB DirectMap1G: 502267904 kB [stack@localhost devstack]$
</DL>
Build is the latest git clone of Devstack.
Thanks
David
Hi Sean
I've set OVS_NUM_HUGEPAGES=14336 but now Devstack is failing to install...
On Wed, 2019-01-30 at 19:02 +0000, David Lake wrote: that appars to be unrelated you could disable the installation of tempest as a workaround but my guess is that it is related to the pip 19.0 or 19.0.1 relsase that was don in the last few days https://pypi.org/project/pip/#history pip config was intoduced in pip 10.0.0b1 https://pip.pypa.io/en/stable/news/#b1-2018-03-31 to disable tempest add "disable_service tempest" to your local.conf then unstack and stack.
David
full create: /opt/stack/tempest/.tox/tempest ERROR: invocation failed (exit code 1), logfile: /opt/stack/tempest/.tox/tempest/log/full-0.log ERROR: actionid: full msg: getenv cmdargs: '/usr/bin/python -m virtualenv --python /usr/bin/python tempest'
Already using interpreter /usr/bin/python New python executable in /opt/stack/tempest/.tox/tempest/bin/python Complete output from command /opt/stack/tempest/.tox/tempest/bin/python -m pip config list: ERROR: unknown command "config" ---------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/virtualenv.py", line 2502, in <module> main() File "/usr/lib/python2.7/site-packages/virtualenv.py", line 793, in main symlink=options.symlink, File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1087, in create_environment install_wheel(to_install, py_executable, search_dirs, download=download) File "/usr/lib/python2.7/site-packages/virtualenv.py", line 935, in install_wheel _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs) File "/usr/lib/python2.7/site-packages/virtualenv.py", line 964, in _install_wheel_with_search_dir config = _pip_config(py_executable, python_path) File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1038, in _pip_config remove_from_env=["PIP_VERBOSE", "PIP_QUIET"], File "/usr/lib/python2.7/site-packages/virtualenv.py", line 886, in call_subprocess raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode)) OSError: Command /opt/stack/tempest/.tox/tempest/bin/python -m pip config list failed with error code 1
ERROR: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python -m virtualenv --python /usr/bin/python tempest (see /opt/stack/tempest/.tox/tempest/log/full-0.log)', 1) ___________________________________ summary ____________________________________ ERROR: full: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python -m virtualenv --python /usr/bin/python tempest (see /opt/stack/tempest/.tox/tempest/log/full-0.log)', 1)
-----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: 29 January 2019 21:46 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack-dev@lists.openstack.org Cc: Ge, Chang Dr (Elec Electronic Eng) <c.ge@surrey.ac.uk> Subject: Re: Issue with launching instance with OVS-DPDK
On Tue, 2019-01-29 at 18:05 +0000, David Lake wrote:
Answers <DL> in-line </DL>
Thanks
David
-----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: 29 January 2019 14:55 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack-dev@lists.openstack.org Cc: Ge, Chang Dr (Elec Electronic Eng) <c.ge@surrey.ac.uk> Subject: Re: Issue with launching instance with OVS-DPDK
On Mon, 2019-01-28 at 13:17 +0000, David Lake wrote:
Hello
I’ve built an Openstack all-in-one using OVS-DPDK via Devstack.
I can launch instances which use the “m1.small” flavour (which I have modified to include the hw:mem_size large as per the DPDK instructions) but as soon as I try to launch anything more than m1.small, I get this error:
Jan 28 12:56:52 localhost nova-conductor: #033[01;31mERROR nova.scheduler.utils [#033[01;36mNone req-917cd3b9-8ce6- 41af-8d44-045002512c91 #033[00;36madmin admin#033[01;31m] #033[01;35m[instance: 25cfee28-08e9-419c-afdb-4d0fe515fb2a] #033[01;31mError from last host: localhost (node localhost): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1935, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2215, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 25cfee28-08e9- 419c-afdb-4d0fe515fb2a was re-scheduled: internal error: qemu unexpectedly closed the monitor: 2019-01- 28T12:56:48.127594Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu46b3c508-f8,server: info: QEMU waiting for connection on: disconnected:unix:/var/run/openvswitch/vhu46b3c508-f8,server\n2019-0 1- 28T12:56:49.251071Z qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepage s/ libvirt/qemu/4-instance- 00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m
My Hypervisor is reporting 510.7GB of RAM and 61 vCPUs.
how much of that ram did you allocate as hugepages.
<DL> OVS_NUM_HUGEPAGES=3072 </DL>
ok so you used networking-ovs-dpdks ablitiy to automatically allocate 2MB hugepages at runtime so this should have allocate 6GB of hugepages per numa node.
can you provide the output of cat /proc/meminfo
<DL>
MemTotal: 526779552 kB MemFree: 466555316 kB MemAvailable: 487218548 kB Buffers: 2308 kB Cached: 22962972 kB SwapCached: 0 kB Active: 29493384 kB Inactive: 13344640 kB Active(anon): 20826364 kB Inactive(anon): 522012 kB Active(file): 8667020 kB Inactive(file): 12822628 kB Unevictable: 43636 kB Mlocked: 47732 kB SwapTotal: 4194300 kB SwapFree: 4194300 kB Dirty: 20 kB Writeback: 0 kB AnonPages: 19933028 kB Mapped: 171680 kB Shmem: 1450564 kB Slab: 1224444 kB SReclaimable: 827696 kB SUnreclaim: 396748 kB KernelStack: 69392 kB PageTables: 181020 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 261292620 kB Committed_AS: 84420252 kB VmallocTotal: 34359738367 kB VmallocUsed: 1352128 kB VmallocChunk: 34154915836 kB HardwareCorrupted: 0 kB AnonHugePages: 5365760 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 6144
since we have 6144 total and OVS_NUM_HUGEPAGES was set to 3072 this indicate the host has 2 numa nodes
HugePages_Free: 2048
and you currently have 4G of 2MB hugepages free. however this will also be split across numa nodes.
the qemu commandline you provied which i have coppied below is trying to allocate 4G of hugepage memory from a single host numa node
qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/ libvirt/qemu/4-instance- 00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m
as a result the vm is failing to boot because nova cannot create the vm with a singel numa node.
if you set hw:numa_nodes=2 this vm would likely boot but since you have a 512G hostyou should be able to increase OVS_NUM_HUGEPAGES to something like OVS_NUM_HUGEPAGES=14336. this will allocate 60G of 2MB hugepages total.
if you want to allocate more then about 96G of hugepages you should set OVS_ALLOCATE_HUGEPAGES=False and instead allcoate the hugepages on the kernel commandline using 1G hugepages. e.g. default_hugepagesz=1G hugepagesz=1G hugepages=480 This is becase it take a long time for ovs-dpdk to scan all the hugepages on start up.
setting default_hugepagesz=1G hugepagesz=1G hugepages=480 will leave 32G of ram for the host. if it a comptue node and not a contorller you can safly reduce the the free host ram to 16G e.g. default_hugepagesz=1G hugepagesz=1G hugepages=496 i would not advice allocating much more above than 496G of hugepages as the qemu emularot over head can eaially get into the 10s of gigs if you have 50+ vms running.
HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 746304 kB DirectMap2M: 34580480 kB DirectMap1G: 502267904 kB [stack@localhost devstack]$
</DL>
Build is the latest git clone of Devstack.
Thanks
David
Hi Sean
I've set OVS_NUM_HUGEPAGES=14336 but now Devstack is failing to install...
Hi Sean Thanks! Got it working and I can now spin up larger VMs. All I've got to do now is work out how to get SSSE3 support in my VM. I think I need to modify the flavour to "Haswell" for that? David -----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: 30 January 2019 19:58 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack-dev@lists.openstack.org Cc: Ge, Chang Dr (Elec Electronic Eng) <c.ge@surrey.ac.uk> Subject: Re: Issue with launching instance with OVS-DPDK On Wed, 2019-01-30 at 19:02 +0000, David Lake wrote: that appars to be unrelated you could disable the installation of tempest as a workaround but my guess is that it is related to the pip 19.0 or 19.0.1 relsase that was don in the last few days https://pypi.org/project/pip/#history pip config was intoduced in pip 10.0.0b1 https://pip.pypa.io/en/stable/news/#b1-2018-03-31 to disable tempest add "disable_service tempest" to your local.conf then unstack and stack.
David
full create: /opt/stack/tempest/.tox/tempest ERROR: invocation failed (exit code 1), logfile: /opt/stack/tempest/.tox/tempest/log/full-0.log ERROR: actionid: full msg: getenv cmdargs: '/usr/bin/python -m virtualenv --python /usr/bin/python tempest'
Already using interpreter /usr/bin/python New python executable in /opt/stack/tempest/.tox/tempest/bin/python Complete output from command /opt/stack/tempest/.tox/tempest/bin/python -m pip config list: ERROR: unknown command "config" ---------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/virtualenv.py", line 2502, in <module> main() File "/usr/lib/python2.7/site-packages/virtualenv.py", line 793, in main symlink=options.symlink, File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1087, in create_environment install_wheel(to_install, py_executable, search_dirs, download=download) File "/usr/lib/python2.7/site-packages/virtualenv.py", line 935, in install_wheel _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs) File "/usr/lib/python2.7/site-packages/virtualenv.py", line 964, in _install_wheel_with_search_dir config = _pip_config(py_executable, python_path) File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1038, in _pip_config remove_from_env=["PIP_VERBOSE", "PIP_QUIET"], File "/usr/lib/python2.7/site-packages/virtualenv.py", line 886, in call_subprocess raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode)) OSError: Command /opt/stack/tempest/.tox/tempest/bin/python -m pip config list failed with error code 1
ERROR: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python -m virtualenv --python /usr/bin/python tempest (see /opt/stack/tempest/.tox/tempest/log/full-0.log)', 1) ___________________________________ summary ____________________________________ ERROR: full: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python -m virtualenv --python /usr/bin/python tempest (see /opt/stack/tempest/.tox/tempest/log/full-0.log)', 1)
-----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: 29 January 2019 21:46 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack-dev@lists.openstack.org Cc: Ge, Chang Dr (Elec Electronic Eng) <c.ge@surrey.ac.uk> Subject: Re: Issue with launching instance with OVS-DPDK
On Tue, 2019-01-29 at 18:05 +0000, David Lake wrote:
Answers <DL> in-line </DL>
Thanks
David
-----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: 29 January 2019 14:55 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack-dev@lists.openstack.org Cc: Ge, Chang Dr (Elec Electronic Eng) <c.ge@surrey.ac.uk> Subject: Re: Issue with launching instance with OVS-DPDK
On Mon, 2019-01-28 at 13:17 +0000, David Lake wrote:
Hello
I’ve built an Openstack all-in-one using OVS-DPDK via Devstack.
I can launch instances which use the “m1.small” flavour (which I have modified to include the hw:mem_size large as per the DPDK instructions) but as soon as I try to launch anything more than m1.small, I get this error:
Jan 28 12:56:52 localhost nova-conductor: #033[01;31mERROR nova.scheduler.utils [#033[01;36mNone req-917cd3b9-8ce6- 41af-8d44-045002512c91 #033[00;36madmin admin#033[01;31m] #033[01;35m[instance: 25cfee28-08e9-419c-afdb-4d0fe515fb2a] #033[01;31mError from last host: localhost (node localhost): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1935, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2215, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 25cfee28-08e9- 419c-afdb-4d0fe515fb2a was re-scheduled: internal error: qemu unexpectedly closed the monitor: 2019-01- 28T12:56:48.127594Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu46b3c508-f8,server: info: QEMU waiting for connection on: disconnected:unix:/var/run/openvswitch/vhu46b3c508-f8,server\n2019 -0 1- 28T12:56:49.251071Z qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepa ge s/ libvirt/qemu/4-instance- 00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m
My Hypervisor is reporting 510.7GB of RAM and 61 vCPUs.
how much of that ram did you allocate as hugepages.
<DL> OVS_NUM_HUGEPAGES=3072 </DL>
ok so you used networking-ovs-dpdks ablitiy to automatically allocate 2MB hugepages at runtime so this should have allocate 6GB of hugepages per numa node.
can you provide the output of cat /proc/meminfo
<DL>
MemTotal: 526779552 kB MemFree: 466555316 kB MemAvailable: 487218548 kB Buffers: 2308 kB Cached: 22962972 kB SwapCached: 0 kB Active: 29493384 kB Inactive: 13344640 kB Active(anon): 20826364 kB Inactive(anon): 522012 kB Active(file): 8667020 kB Inactive(file): 12822628 kB Unevictable: 43636 kB Mlocked: 47732 kB SwapTotal: 4194300 kB SwapFree: 4194300 kB Dirty: 20 kB Writeback: 0 kB AnonPages: 19933028 kB Mapped: 171680 kB Shmem: 1450564 kB Slab: 1224444 kB SReclaimable: 827696 kB SUnreclaim: 396748 kB KernelStack: 69392 kB PageTables: 181020 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 261292620 kB Committed_AS: 84420252 kB VmallocTotal: 34359738367 kB VmallocUsed: 1352128 kB VmallocChunk: 34154915836 kB HardwareCorrupted: 0 kB AnonHugePages: 5365760 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 6144
since we have 6144 total and OVS_NUM_HUGEPAGES was set to 3072 this indicate the host has 2 numa nodes
HugePages_Free: 2048
and you currently have 4G of 2MB hugepages free. however this will also be split across numa nodes.
the qemu commandline you provied which i have coppied below is trying to allocate 4G of hugepage memory from a single host numa node
qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/ libvirt/qemu/4-instance- 00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m
as a result the vm is failing to boot because nova cannot create the vm with a singel numa node.
if you set hw:numa_nodes=2 this vm would likely boot but since you have a 512G hostyou should be able to increase OVS_NUM_HUGEPAGES to something like OVS_NUM_HUGEPAGES=14336. this will allocate 60G of 2MB hugepages total.
if you want to allocate more then about 96G of hugepages you should set OVS_ALLOCATE_HUGEPAGES=False and instead allcoate the hugepages on the kernel commandline using 1G hugepages. e.g. default_hugepagesz=1G hugepagesz=1G hugepages=480 This is becase it take a long time for ovs-dpdk to scan all the hugepages on start up.
setting default_hugepagesz=1G hugepagesz=1G hugepages=480 will leave 32G of ram for the host. if it a comptue node and not a contorller you can safly reduce the the free host ram to 16G e.g. default_hugepagesz=1G hugepagesz=1G hugepages=496 i would not advice allocating much more above than 496G of hugepages as the qemu emularot over head can eaially get into the 10s of gigs if you have 50+ vms running.
HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 746304 kB DirectMap2M: 34580480 kB DirectMap1G: 502267904 kB [stack@localhost devstack]$
</DL>
Build is the latest git clone of Devstack.
Thanks
David
Hi Sean
Thanks!
Got it working and I can now spin up larger VMs.
All I've got to do now is work out how to get SSSE3 support in my VM. I think I need to modify the flavour to "Haswell" for that?
On Wed, 2019-01-30 at 22:24 +0000, David Lake wrote: there are a few ways but my perfered way is to configre libvirt to use the hosts cpu features. add the following at the very end of your local.conf for future restacks [[post-config|/etc/nova/nova.conf]] [filter_scheduler] enabled_filters = RetryFilter,AvailabilityZoneFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinity Filter,ServerGroupAffinityFilter,SameHostFilter,DifferentHostFilter,PciPassthroughFilter,NUMATopologyFilter [libvirt] cpu_mode = host-passthrough virt_type = kvm [[post-config|$NOVA_CPU_CONF]] [libvirt] cpu_mode = host-passthrough virt_type = kvm in the mean time you can just add [libvirt] cpu_mode = host-passthrough virt_type = kvm to /etc/nova/nova-cpu.conf and restart nova compute sudo systemctl restart devstack@n-cpu with those changes your vms will have acess to all the cpu features of the host cpu.
David -----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: 30 January 2019 19:58 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack-dev@lists.openstack.org Cc: Ge, Chang Dr (Elec Electronic Eng) <c.ge@surrey.ac.uk> Subject: Re: Issue with launching instance with OVS-DPDK On Wed, 2019-01-30 at 19:02 +0000, David Lake wrote: Hi Sean I've set OVS_NUM_HUGEPAGES=14336 but now Devstack is failing to install... that appars to be unrelated you could disable the installation of tempest as a workaround but my guess is that it is related to the pip 19.0 or 19.0.1 relsase that was don in the last few days https://pypi.org/project/pip/#history pip config was intoduced in pip 10.0.0b1 https://pip.pypa.io/en/stable/news/#b1-2018-03-31 to disable tempest add "disable_service tempest" to your local.conf then unstack and stack. David full create: /opt/stack/tempest/.tox/tempest ERROR: invocation failed (exit code 1), logfile: /opt/stack/tempest/.tox/tempest/log/full-0.log ERROR: actionid: full msg: getenv cmdargs: '/usr/bin/python -m virtualenv --python /usr/bin/python tempest' Already using interpreter /usr/bin/python New python executable in /opt/stack/tempest/.tox/tempest/bin/python Complete output from command /opt/stack/tempest/.tox/tempest/bin/python -m pip config list: ERROR: unknown command "config" ---------------------------------------- Traceback (most recent call last): File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main "__main__", fname, loader, pkg_name) File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/usr/lib/python2.7/site-packages/virtualenv.py", line 2502, in <module> main() File "/usr/lib/python2.7/site-packages/virtualenv.py", line 793, in main symlink=options.symlink, File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1087, in create_environment install_wheel(to_install, py_executable, search_dirs, download=download) File "/usr/lib/python2.7/site-packages/virtualenv.py", line 935, in install_wheel _install_wheel_with_search_dir(download, project_names, py_executable, search_dirs) File "/usr/lib/python2.7/site-packages/virtualenv.py", line 964, in _install_wheel_with_search_dir config = _pip_config(py_executable, python_path) File "/usr/lib/python2.7/site-packages/virtualenv.py", line 1038, in _pip_config remove_from_env=["PIP_VERBOSE", "PIP_QUIET"], File "/usr/lib/python2.7/site-packages/virtualenv.py", line 886, in call_subprocess raise OSError("Command {} failed with error code {}".format(cmd_desc, proc.returncode)) OSError: Command /opt/stack/tempest/.tox/tempest/bin/python -m pip config list failed with error code 1 ERROR: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python -m virtualenv --python /usr/bin/python tempest (see /opt/stack/tempest/.tox/tempest/log/full-0.log)', 1) ___________________________________ summary ____________________________________ ERROR: full: Error creating virtualenv. Note that some special characters (e.g. ':' and unicode symbols) in paths are not supported by virtualenv. Error details: InvocationError('/usr/bin/python -m virtualenv --python /usr/bin/python tempest (see /opt/stack/tempest/.tox/tempest/log/full-0.log)', 1) -----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: 29 January 2019 21:46 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack-dev@lists.openstack.org Cc: Ge, Chang Dr (Elec Electronic Eng) <c.ge@surrey.ac.uk> Subject: Re: Issue with launching instance with OVS-DPDK On Tue, 2019-01-29 at 18:05 +0000, David Lake wrote: Answers <DL> in-line </DL> Thanks David -----Original Message----- From: Sean Mooney <smooney@redhat.com> Sent: 29 January 2019 14:55 To: Lake, David (PG/R - Elec Electronic Eng) <d.lake@surrey.ac.uk>; openstack-dev@lists.openstack.org Cc: Ge, Chang Dr (Elec Electronic Eng) <c.ge@surrey.ac.uk> Subject: Re: Issue with launching instance with OVS-DPDK On Mon, 2019-01-28 at 13:17 +0000, David Lake wrote: Hello I’ve built an Openstack all-in-one using OVS-DPDK via Devstack. I can launch instances which use the “m1.small” flavour (which I have modified to include the hw:mem_size large as per the DPDK instructions) but as soon as I try to launch anything more than m1.small, I get this error: Jan 28 12:56:52 localhost nova-conductor: #033[01;31mERROR nova.scheduler.utils [#033[01;36mNone req-917cd3b9-8ce6- 41af-8d44-045002512c91 #033[00;36madmin admin#033[01;31m] #033[01;35m[instance: 25cfee28-08e9-419c-afdb-4d0fe515fb2a] #033[01;31mError from last host: localhost (node localhost): [u'Traceback (most recent call last):\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 1935, in _do_build_and_run_instance\n filter_properties, request_spec)\n', u' File "/opt/stack/nova/nova/compute/manager.py", line 2215, in _build_and_run_instance\n instance_uuid=instance.uuid, reason=six.text_type(e))\n', u'RescheduledException: Build of instance 25cfee28-08e9- 419c-afdb-4d0fe515fb2a was re-scheduled: internal error: qemu unexpectedly closed the monitor: 2019-01- 28T12:56:48.127594Z qemu-kvm: -chardev socket,id=charnet0,path=/var/run/openvswitch/vhu46b3c508-f8,server: info: QEMU waiting for connection on: disconnected:unix:/var/run/openvswitch/vhu46b3c508-f8,server\n2019 -0 1- 28T12:56:49.251071Z qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepa ge s/ libvirt/qemu/4-instance- 00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m My Hypervisor is reporting 510.7GB of RAM and 61 vCPUs. how much of that ram did you allocate as hugepages. <DL> OVS_NUM_HUGEPAGES=3072 </DL> ok so you used networking-ovs-dpdks ablitiy to automatically allocate 2MB hugepages at runtime so this should have allocate 6GB of hugepages per numa node. can you provide the output of cat /proc/meminfo <DL> MemTotal: 526779552 kB MemFree: 466555316 kB MemAvailable: 487218548 kB Buffers: 2308 kB Cached: 22962972 kB SwapCached: 0 kB Active: 29493384 kB Inactive: 13344640 kB Active(anon): 20826364 kB Inactive(anon): 522012 kB Active(file): 8667020 kB Inactive(file): 12822628 kB Unevictable: 43636 kB Mlocked: 47732 kB SwapTotal: 4194300 kB SwapFree: 4194300 kB Dirty: 20 kB Writeback: 0 kB AnonPages: 19933028 kB Mapped: 171680 kB Shmem: 1450564 kB Slab: 1224444 kB SReclaimable: 827696 kB SUnreclaim: 396748 kB KernelStack: 69392 kB PageTables: 181020 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 261292620 kB Committed_AS: 84420252 kB VmallocTotal: 34359738367 kB VmallocUsed: 1352128 kB VmallocChunk: 34154915836 kB HardwareCorrupted: 0 kB AnonHugePages: 5365760 kB CmaTotal: 0 kB CmaFree: 0 kB HugePages_Total: 6144 since we have 6144 total and OVS_NUM_HUGEPAGES was set to 3072 this indicate the host has 2 numa nodes HugePages_Free: 2048 and you currently have 4G of 2MB hugepages free. however this will also be split across numa nodes. the qemu commandline you provied which i have coppied below is trying to allocate 4G of hugepage memory from a single host numa node qemu-kvm: -object memory-backend-file,id=ram-node0,prealloc=yes,mem-path=/dev/hugepages/ libvirt/qemu/4-instance- 00000005,share=yes,size=4294967296,host-nodes=0,policy=bind: os_mem_prealloc: Insufficient free host memory pages available to allocate guest RAM\n']#033[00m#033[00m as a result the vm is failing to boot because nova cannot create the vm with a singel numa node. if you set hw:numa_nodes=2 this vm would likely boot but since you have a 512G hostyou should be able to increase OVS_NUM_HUGEPAGES to something like OVS_NUM_HUGEPAGES=14336. this will allocate 60G of 2MB hugepages total. if you want to allocate more then about 96G of hugepages you should set OVS_ALLOCATE_HUGEPAGES=False and instead allcoate the hugepages on the kernel commandline using 1G hugepages. e.g. default_hugepagesz=1G hugepagesz=1G hugepages=480 This is becase it take a long time for ovs-dpdk to scan all the hugepages on start up. setting default_hugepagesz=1G hugepagesz=1G hugepages=480 will leave 32G of ram for the host. if it a comptue node and not a contorller you can safly reduce the the free host ram to 16G e.g. default_hugepagesz=1G hugepagesz=1G hugepages=496 i would not advice allocating much more above than 496G of hugepages as the qemu emularot over head can eaially get into the 10s of gigs if you have 50+ vms running. HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 2048 kB DirectMap4k: 746304 kB DirectMap2M: 34580480 kB DirectMap1G: 502267904 kB [stack@localhost devstack]$ </DL> Build is the latest git clone of Devstack. Thanks David
On 01/30/2019 02:58 PM, Sean Mooney wrote:
Hi Sean
I've set OVS_NUM_HUGEPAGES=14336 but now Devstack is failing to install...
On Wed, 2019-01-30 at 19:02 +0000, David Lake wrote: that appars to be unrelated you could disable the installation of tempest as a workaround but my guess is that it is related to the pip 19.0 or 19.0.1 relsase that was don in the last few days https://pypi.org/project/pip/#history
pip config was intoduced in pip 10.0.0b1 https://pip.pypa.io/en/stable/news/#b1-2018-03-31
On Wed, 2019-01-30 at 17:41 -0500, Jay Pipes wrote:
On 01/30/2019 02:58 PM, Sean Mooney wrote:
On Wed, 2019-01-30 at 19:02 +0000, David Lake wrote:
Hi Sean
I've set OVS_NUM_HUGEPAGES=14336 but now Devstack is failing to install...
that appars to be unrelated you could disable the installation of tempest as a workaround but my guess is that it is related to the pip 19.0 or 19.0.1 relsase that was don in the last few days https://pypi.org/project/pip/#history
pip config was intoduced in pip 10.0.0b1 https://pip.pypa.io/en/stable/news/#b1-2018-03-31
ah yes i completely forgot about the version cap in devstack that makes sense good catch.
-jay
participants (3)
-
David Lake
-
Jay Pipes
-
Sean Mooney