Hi all, Just reading over the docs describing NUMA scheduler filter testing (http://docs.openstack.org/developer/nova/devref/testing/libvirt-numa.html#te...). Somewhat confused by the scenario at the end, which seems to call for a guest with 1 NUMA node but looks like it gets created with 2... I'm obviously misunderstanding something in the resultant XML, but there's no topology info shown from inside the guest, so I'm not sure. Anyone tried this? -- Cheers, ~Blairo
I think it creates 2 Virtual NUMA nodes on one single Physical NUMA node If you see below cpu 0-1 are in one NUMA node and 2-3 are in the other. This is what I can understand from the xml. <cell id='0' cpus='0-1' memory='524288'/> <cell id='1' cpus='2-3' memory='524288'/ On 29 April 2015 at 10:36, Blair Bethwaite <blair.bethwaite@gmail.com> wrote:
Hi all,
Just reading over the docs describing NUMA scheduler filter testing ( http://docs.openstack.org/developer/nova/devref/testing/libvirt-numa.html#te... ). Somewhat confused by the scenario at the end, which seems to call for a guest with 1 NUMA node but looks like it gets created with 2... I'm obviously misunderstanding something in the resultant XML, but there's no topology info shown from inside the guest, so I'm not sure. Anyone tried this?
-- Cheers, ~Blairo
_______________________________________________ OpenStack-HPC mailing list OpenStack-HPC@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc
On 29 April 2015 at 18:50, Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:
I think it creates 2 Virtual NUMA nodes on one single Physical NUMA node If you see below cpu 0-1 are in one NUMA node and 2-3 are in the other. This is what I can understand from the xml.
<numa> <cell id='0' cpus='0-1' memory='524288'/> <cell id='1' cpus='2-3' memory='524288'/> </numa>
Yeah, but that doesn't seem to be consistent with this from the same section: "nova flavor-key m1.numa set hw:numa_nodes=1" According to the spec (http://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/virt-...): hw:numa_nodes=NN - numa of NUMA nodes to expose to the guest. Not to mention the numatune config that binds the guest nodes to host nodes. An aside: Does anyone else find it strange that all these options are designed to be flavor and/or image extra specs rather than providing a mechanism to set them on instance boot (e.g. hints). I think the relationship between flavors or images and such tunables is tenuous at best, why should I need multiple versions of what is otherwise the same image or flavor in order to ask for e.g. CPU pinning or Qemu guest agent, these are per instance tunables. -- Cheers, ~Blairo
Adding Dan and Nikola as I doubt they are on this list, guys this is in reference to this devref example which looks a little off: http://docs.openstack.org/developer/nova/devref/testing/libvirt-numa.html#te... ----- Original Message -----
From: "Blair Bethwaite" <blair.bethwaite@gmail.com> To: "Pradeep Kiruvale" <pradeepkiruvale@gmail.com> Cc: openstack-hpc@lists.openstack.org Sent: Wednesday, April 29, 2015 10:33:31 AM Subject: Re: [openstack-hpc] NUMA config
On 29 April 2015 at 18:50, Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:
I think it creates 2 Virtual NUMA nodes on one single Physical NUMA node If you see below cpu 0-1 are in one NUMA node and 2-3 are in the other. This is what I can understand from the xml.
<numa> <cell id='0' cpus='0-1' memory='524288'/> <cell id='1' cpus='2-3' memory='524288'/> </numa>
Yeah, but that doesn't seem to be consistent with this from the same section: "nova flavor-key m1.numa set hw:numa_nodes=1" According to the spec (http://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/virt-...): hw:numa_nodes=NN - numa of NUMA nodes to expose to the guest. Not to mention the numatune config that binds the guest nodes to host nodes.
An aside: Does anyone else find it strange that all these options are designed to be flavor and/or image extra specs rather than providing a mechanism to set them on instance boot (e.g. hints). I think the relationship between flavors or images and such tunables is tenuous at best, why should I need multiple versions of what is otherwise the same image or flavor in order to ask for e.g. CPU pinning or Qemu guest agent, these are per instance tunables.
-- Cheers, ~Blairo
_______________________________________________ OpenStack-HPC mailing list OpenStack-HPC@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc
-- Steve Gordon, RHCE Sr. Technical Product Manager, Red Hat Enterprise Linux OpenStack Platform
On Wed, Apr 29, 2015 at 11:12:54AM -0400, Steve Gordon wrote:
Adding Dan and Nikola as I doubt they are on this list, guys this is in reference to this devref example which looks a little off:
http://docs.openstack.org/developer/nova/devref/testing/libvirt-numa.html#te...
Yes, sorry my bad. That XML example is wrong - it is the XML you'd see from a hw:numa_nodes=2 config, not hw:numa_nodes=1
----- Original Message -----
From: "Blair Bethwaite" <blair.bethwaite@gmail.com> To: "Pradeep Kiruvale" <pradeepkiruvale@gmail.com> Cc: openstack-hpc@lists.openstack.org Sent: Wednesday, April 29, 2015 10:33:31 AM Subject: Re: [openstack-hpc] NUMA config
On 29 April 2015 at 18:50, Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:
I think it creates 2 Virtual NUMA nodes on one single Physical NUMA node If you see below cpu 0-1 are in one NUMA node and 2-3 are in the other. This is what I can understand from the xml.
<numa> <cell id='0' cpus='0-1' memory='524288'/> <cell id='1' cpus='2-3' memory='524288'/> </numa>
Yeah, but that doesn't seem to be consistent with this from the same section: "nova flavor-key m1.numa set hw:numa_nodes=1" According to the spec (http://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/virt-...): hw:numa_nodes=NN - numa of NUMA nodes to expose to the guest. Not to mention the numatune config that binds the guest nodes to host nodes.
An aside: Does anyone else find it strange that all these options are designed to be flavor and/or image extra specs rather than providing a mechanism to set them on instance boot (e.g. hints). I think the relationship between flavors or images and such tunables is tenuous at best, why should I need multiple versions of what is otherwise the same image or flavor in order to ask for e.g. CPU pinning or Qemu guest agent, these are per instance tunables.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Thanks Daniel, I'll fix that, need to try my hand at doc patches. Whilst I've got your attention, any comment on my aside about it seeming odd from userland perspective that these (and similar tunables) are controlled through flavor/image extra specs? "Does anyone else find it strange that all these options are designed to be flavor and/or image extra specs rather than providing a mechanism to set them on instance boot (e.g. hints). I think the relationship between flavors or images and such tunables is tenuous at best, why should I need multiple versions of what is otherwise the same image or flavor in order to ask for e.g. CPU pinning or Qemu Guest Agent, these are per instance tunables/variables." Cheers, On 30 April 2015 at 01:31, Daniel P. Berrange <berrange@redhat.com> wrote:
On Wed, Apr 29, 2015 at 11:12:54AM -0400, Steve Gordon wrote:
Adding Dan and Nikola as I doubt they are on this list, guys this is in reference to this devref example which looks a little off:
http://docs.openstack.org/developer/nova/devref/testing/libvirt-numa.html#te...
Yes, sorry my bad. That XML example is wrong - it is the XML you'd see from a hw:numa_nodes=2 config, not hw:numa_nodes=1
----- Original Message -----
From: "Blair Bethwaite" <blair.bethwaite@gmail.com> To: "Pradeep Kiruvale" <pradeepkiruvale@gmail.com> Cc: openstack-hpc@lists.openstack.org Sent: Wednesday, April 29, 2015 10:33:31 AM Subject: Re: [openstack-hpc] NUMA config
On 29 April 2015 at 18:50, Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:
I think it creates 2 Virtual NUMA nodes on one single Physical NUMA node If you see below cpu 0-1 are in one NUMA node and 2-3 are in the other. This is what I can understand from the xml.
<numa> <cell id='0' cpus='0-1' memory='524288'/> <cell id='1' cpus='2-3' memory='524288'/> </numa>
Yeah, but that doesn't seem to be consistent with this from the same section: "nova flavor-key m1.numa set hw:numa_nodes=1" According to the spec (http://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/virt-...): hw:numa_nodes=NN - numa of NUMA nodes to expose to the guest. Not to mention the numatune config that binds the guest nodes to host nodes.
An aside: Does anyone else find it strange that all these options are designed to be flavor and/or image extra specs rather than providing a mechanism to set them on instance boot (e.g. hints). I think the relationship between flavors or images and such tunables is tenuous at best, why should I need multiple versions of what is otherwise the same image or flavor in order to ask for e.g. CPU pinning or Qemu guest agent, these are per instance tunables.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- Cheers, ~Blairo
We have had similar debates for the host passthrough and disk cache options in Nova which are set in nova.conf. We find that the application launcher should determine this (may be subject to an administrator override). Options like guest agent seem reasonable to be on the image (since they need code). Tim On 4/30/15, 3:21 AM, "Blair Bethwaite" <blair.bethwaite@gmail.com> wrote:
Thanks Daniel,
I'll fix that, need to try my hand at doc patches.
Whilst I've got your attention, any comment on my aside about it seeming odd from userland perspective that these (and similar tunables) are controlled through flavor/image extra specs? "Does anyone else find it strange that all these options are designed to be flavor and/or image extra specs rather than providing a mechanism to set them on instance boot (e.g. hints). I think the relationship between flavors or images and such tunables is tenuous at best, why should I need multiple versions of what is otherwise the same image or flavor in order to ask for e.g. CPU pinning or Qemu Guest Agent, these are per instance tunables/variables."
Cheers,
On 30 April 2015 at 01:31, Daniel P. Berrange <berrange@redhat.com> wrote:
On Wed, Apr 29, 2015 at 11:12:54AM -0400, Steve Gordon wrote:
Adding Dan and Nikola as I doubt they are on this list, guys this is in reference to this devref example which looks a little off:
http://docs.openstack.org/developer/nova/devref/testing/libvirt-numa.htm l#testing-instance-boot-with-1-numa-cell-requested
Yes, sorry my bad. That XML example is wrong - it is the XML you'd see from a hw:numa_nodes=2 config, not hw:numa_nodes=1
----- Original Message -----
From: "Blair Bethwaite" <blair.bethwaite@gmail.com> To: "Pradeep Kiruvale" <pradeepkiruvale@gmail.com> Cc: openstack-hpc@lists.openstack.org Sent: Wednesday, April 29, 2015 10:33:31 AM Subject: Re: [openstack-hpc] NUMA config
On 29 April 2015 at 18:50, Pradeep Kiruvale <pradeepkiruvale@gmail.com> wrote:
I think it creates 2 Virtual NUMA nodes on one single Physical NUMA node If you see below cpu 0-1 are in one NUMA node and 2-3 are in the other. This is what I can understand from the xml.
<numa> <cell id='0' cpus='0-1' memory='524288'/> <cell id='1' cpus='2-3' memory='524288'/> </numa>
Yeah, but that doesn't seem to be consistent with this from the same section: "nova flavor-key m1.numa set hw:numa_nodes=1" According to the spec
(http://specs.openstack.org/openstack/nova-specs/specs/juno/implemented/ virt-driver-numa-placement.html):
hw:numa_nodes=NN - numa of NUMA nodes to expose to the guest. Not to mention the numatune config that binds the guest nodes to host nodes.
An aside: Does anyone else find it strange that all these options are designed to be flavor and/or image extra specs rather than providing a mechanism to set them on instance boot (e.g. hints). I think the relationship between flavors or images and such tunables is tenuous at best, why should I need multiple versions of what is otherwise the same image or flavor in order to ask for e.g. CPU pinning or Qemu guest agent, these are per instance tunables.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- Cheers, ~Blairo
_______________________________________________ OpenStack-HPC mailing list OpenStack-HPC@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc
On Thu, Apr 30, 2015 at 11:21:26AM +1000, Blair Bethwaite wrote:
Thanks Daniel,
I'll fix that, need to try my hand at doc patches.
I've actually sent a patch for that late last night already https://review.openstack.org/#/c/178773/
Whilst I've got your attention, any comment on my aside about it seeming odd from userland perspective that these (and similar tunables) are controlled through flavor/image extra specs? "Does anyone else find it strange that all these options are designed to be flavor and/or image extra specs rather than providing a mechanism to set them on instance boot (e.g. hints). I think the relationship between flavors or images and such tunables is tenuous at best, why should I need multiple versions of what is otherwise the same image or flavor in order to ask for e.g. CPU pinning or Qemu Guest Agent, these are per instance tunables/variables."
Ok, so first of all, NUMA, strict cpu pinning and huge page configuration policies have a direct impact on compute host utilization. The cloud admin has to setup host aggregates to make proper use of dedicated cpu pinning & huge pages, and these imply no use of over commit. Given that CPU & RAM Is a finite resource, the cloud admin must retain a way to prevent users from accessing cpu pinning & huge pages features. The only mechanism to prevent such usage is to set properties on the flavour - the flavours are the core access control mechanism in Nova for resource usage. So the cloud admin can setup flavours which allow or deny use of these features. This is what I'd expect to be the usage pattern for public cloud where permissioning is very important. To make it more useful for private cloud though, we allow all the properties to be set at the image level, /provided/ they don't conflict with anything set at the flavour level. ie the flavour settings always take priority, to prevent tenants violating cloud admin's controls. That is how things stand today. There *is* however a more general goal to enhance the Nova boot API, so that any image property can also be specified at time you boot the instance. This will apply not just for NUMA/CPU/hugepags things, but for absolutely any setting that is currently valid against the image. Before we can do that though, Nova people wanted us to better specify exactly what we support for image properties To that end I've actually been working on a change to make a formally defined object listing all known image properties: https://review.openstack.org/#/c/76234/28/nova/objects/image_meta_props.py Once we get this approved, then we can look at the enhancement for the nova boot API. I'm not going to promise this will happen in Liberty release cycle, but it /is/ something we want to get done for Nova. Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Daniel, Thank you for the detailed response. I take your point on the settings that have a direct effect on resource utilisation etc, and thanks for the info about the ongoing work! Cheers, On 30 April 2015 at 19:38, Daniel P. Berrange <berrange@redhat.com> wrote:
On Thu, Apr 30, 2015 at 11:21:26AM +1000, Blair Bethwaite wrote:
Thanks Daniel,
I'll fix that, need to try my hand at doc patches.
I've actually sent a patch for that late last night already
https://review.openstack.org/#/c/178773/
Whilst I've got your attention, any comment on my aside about it seeming odd from userland perspective that these (and similar tunables) are controlled through flavor/image extra specs? "Does anyone else find it strange that all these options are designed to be flavor and/or image extra specs rather than providing a mechanism to set them on instance boot (e.g. hints). I think the relationship between flavors or images and such tunables is tenuous at best, why should I need multiple versions of what is otherwise the same image or flavor in order to ask for e.g. CPU pinning or Qemu Guest Agent, these are per instance tunables/variables."
Ok, so first of all, NUMA, strict cpu pinning and huge page configuration policies have a direct impact on compute host utilization. The cloud admin has to setup host aggregates to make proper use of dedicated cpu pinning & huge pages, and these imply no use of over commit. Given that CPU & RAM Is a finite resource, the cloud admin must retain a way to prevent users from accessing cpu pinning & huge pages features.
The only mechanism to prevent such usage is to set properties on the flavour - the flavours are the core access control mechanism in Nova for resource usage. So the cloud admin can setup flavours which allow or deny use of these features. This is what I'd expect to be the usage pattern for public cloud where permissioning is very important.
To make it more useful for private cloud though, we allow all the properties to be set at the image level, /provided/ they don't conflict with anything set at the flavour level. ie the flavour settings always take priority, to prevent tenants violating cloud admin's controls.
That is how things stand today. There *is* however a more general goal to enhance the Nova boot API, so that any image property can also be specified at time you boot the instance. This will apply not just for NUMA/CPU/hugepags things, but for absolutely any setting that is currently valid against the image.
Before we can do that though, Nova people wanted us to better specify exactly what we support for image properties To that end I've actually been working on a change to make a formally defined object listing all known image properties:
https://review.openstack.org/#/c/76234/28/nova/objects/image_meta_props.py
Once we get this approved, then we can look at the enhancement for the nova boot API. I'm not going to promise this will happen in Liberty release cycle, but it /is/ something we want to get done for Nova.
Regards, Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
-- Cheers, ~Blairo
participants (5)
-
Blair Bethwaite
-
Daniel P. Berrange
-
Pradeep Kiruvale
-
Steve Gordon
-
Tim Bell