<div dir="ltr">See PCM inline...<br><br><div class="gmail_quote"><div dir="ltr">On Fri, Jun 3, 2016 at 8:44 AM Daniel P. Berrange <<a href="mailto:berrange@redhat.com">berrange@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On Fri, Jun 03, 2016 at 12:32:17PM +0000, Paul Michali wrote:<br>
> Hi!<br>
><br>
> I've been playing with Liberty code a bit and had some questions that I'm<br>
> hoping Nova folks may be able to provide guidance on...<br>
><br>
> If I set up a flavor with hw:mem_page_size=2048, and I'm creating (Cirros)<br>
> VMs with size 1024, will the scheduling use the minimum of the number of<br>
<br>
1024 what units ? 1024 MB, or 1024 huge pages aka 2048 MB ?<br></blockquote><div><br></div><div>PCM: I was using small flavor, which is 2 GB. So that's 2048 MB and the page size is 2048K, so 1024 pages? Hope I have the units right.</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> huge pages available and the size requested for the VM, or will it base<br>
> scheduling only on the number of huge pages?<br>
><br>
> It seems to be doing the latter, where I had 1945 huge pages free, and<br>
> tried to create another VM (1024) and Nova rejected the request with "no<br>
> hosts available".<br>
<br>
>From this I'm guessing you're meaning 1024 huge pages aka 2 GB earlier.<br>
<br>
Anyway, when you request huge pages to be used for a flavour, the<br>
entire guest RAM must be able to be allocated from huge pages.<br>
ie if you have a guest with 2 GB of RAM, you must have 2 GB worth<br>
of huge pages available. It is not possible for a VM to use<br>
1.5 GB of huge pages and 500 MB of normal sized pages.<br></blockquote><div><br></div><div>PCM: Right, so, with 2GB of RAM, I need 1024 huge pages of size 2048K. In this case, there are 1945 huge pages available, so I was wondering why it failed. Maybe I'm confusing sizes/pages?</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> Is this still the same for Mitaka?<br>
<br>
Yep, this use of huge pages has not changed.<br>
<br>
> Where could I look in the code to see how the scheduling is determined?<br>
<br>
Most logic related to huge pages is in nova/virt/hardware.py<br>
<br>
> If I use mem_page_size=large (what I originally had), should it evenly<br>
> assign huge pages from the available NUMA nodes (there are two in my case)?<br>
><br>
> It looks like it was assigning all VMs to the same NUMA node (0) in this<br>
> case. Is the right way to change to 2048, like I did above?<br>
<br>
Nova will always avoid spreading your VM across 2 host NUMA nodes,<br>
since that gives bad performance characteristics. IOW, it will always<br>
allocate huge pages from the NUMA node that the guest will run on. If<br>
you explicitly want your VM to spread across 2 host NUMA nodes, then<br>
you must tell nova to create 2 *guest* NUMA nodes for the VM. Nova<br>
will then place each guest NUMA node, on a separate host NUMA node<br>
and allocate huge pages from node to match. This is done using<br>
the hw:numa_nodes=2 parameter on the flavour<br></blockquote><div><br></div><div>PCM: Gotcha, but that was not the issue I'm seeing. With this small flavor (2GB = 1024 pages), I had 13107 huge pages initially. As I created VMs, they were *all* placed on the same NUMA node (0). As a result, when I got to more than have the available pages, Nova failed to allow further VMs, even though I had 6963 available on one compute node, and 5939 on another.</div><div><br></div><div>It seems that all the assignments were to node zero. Someone suggested to me to set mem_page_size to 2048, and at that point it started assigning to both NUMA nodes evenly.</div><div><br></div><div>Thanks for the help!!!</div><div><br></div><div><br></div><div>Regards,</div><div><br></div><div>PCM</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
> Again, has this changed at all in Mitaka?<br>
<br>
Nope. Well aside from random bug fixes.<br>
<br>
Regards,<br>
Daniel<br>
--<br>
|: <a href="http://berrange.com" rel="noreferrer" target="_blank">http://berrange.com</a> -o- <a href="http://www.flickr.com/photos/dberrange/" rel="noreferrer" target="_blank">http://www.flickr.com/photos/dberrange/</a> :|<br>
|: <a href="http://libvirt.org" rel="noreferrer" target="_blank">http://libvirt.org</a> -o- <a href="http://virt-manager.org" rel="noreferrer" target="_blank">http://virt-manager.org</a> :|<br>
|: <a href="http://autobuild.org" rel="noreferrer" target="_blank">http://autobuild.org</a> -o- <a href="http://search.cpan.org/~danberr/" rel="noreferrer" target="_blank">http://search.cpan.org/~danberr/</a> :|<br>
|: <a href="http://entangle-photo.org" rel="noreferrer" target="_blank">http://entangle-photo.org</a> -o- <a href="http://live.gnome.org/gtk-vnc" rel="noreferrer" target="_blank">http://live.gnome.org/gtk-vnc</a> :|<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div></div>