<div dir="ltr">Awesome Jay!<div><br></div><div>Do you think this is something that can be backporting to Liberty w/o other dependencies? We're running Liberty on our system right now.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jun 14, 2016 at 4:10 PM Jay Pipes <<a href="mailto:jaypipes@gmail.com">jaypipes@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 06/14/2016 12:30 PM, Paul Michali wrote:<br>
> Well, looks like we figured out what is going on - maybe folks have some<br>
> ideas on how we could handle this issue.<br>
><br>
> What I see is that for each VM create (small flavor), 1024 huge pages<br>
> are used and NUMA node 0 used. It appears that, when there is no longer<br>
> enough huge pages on that NUMA node, Nova with then schedule to the<br>
> other NUMA node and use those huge pages.<br>
><br>
> In our case, we happen to have a special container running on the<br>
> compute nodes, that uses 512 huge pages. As a result, when there are 768<br>
> huge pages left, Nova thinks there are 1280 pages left and thinks one<br>
> more VM can be create. It tries, but the create fails.<br>
><br>
> Some questions...<br>
><br>
> 1) Is there some way to "reserve" huge pages in Nova?<br>
<br>
Yes. Code merged recently from Sahid does this:<br>
<br>
<a href="https://review.openstack.org/#/c/277422/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/277422/</a><br>
<br>
Best,<br>
-jay<br>
<br>
> 2) If the create fails, should Nova try the other NUMA node (or is this<br>
> because it doesn't know why it failed)?<br>
> 3) Any ideas on how we can deal with this - without changing Nova?<br>
><br>
> Thanks!<br>
><br>
> PCM<br>
><br>
><br>
><br>
> On Tue, Jun 14, 2016 at 1:09 PM Paul Michali <<a href="mailto:pc@michali.net" target="_blank">pc@michali.net</a><br>
> <mailto:<a href="mailto:pc@michali.net" target="_blank">pc@michali.net</a>>> wrote:<br>
><br>
> Great info Chris and thanks for confirming the assignment of blocks<br>
> of pages to a numa node.<br>
><br>
> I'm still struggling with why each VM is being assigned to NUMA node<br>
> 0. Any ideas on where I should look to see why Nova is not using<br>
> NUMA id 1?<br>
><br>
> Thanks!<br>
><br>
><br>
> PCM<br>
><br>
><br>
> On Tue, Jun 14, 2016 at 10:29 AM Chris Friesen<br>
> <<a href="mailto:chris.friesen@windriver.com" target="_blank">chris.friesen@windriver.com</a> <mailto:<a href="mailto:chris.friesen@windriver.com" target="_blank">chris.friesen@windriver.com</a>>><br>
> wrote:<br>
><br>
> On 06/13/2016 02:17 PM, Paul Michali wrote:<br>
> > Hmm... I tried Friday and again today, and I'm not seeing the<br>
> VMs being evenly<br>
> > created on the NUMA nodes. Every Cirros VM is created on<br>
> nodeid 0.<br>
> ><br>
> > I have the m1/small flavor (@GB) selected and am using<br>
> hw:numa_nodes=1 and<br>
> > hw:mem_page_size=2048 flavor-key settings. Each VM is<br>
> consuming 1024 huge pages<br>
> > (of size 2MB), but is on nodeid 0 always. Also, it seems that<br>
> when I reach 1/2<br>
> > of the total number of huge pages used, libvirt gives an<br>
> error saying there is<br>
> > not enough memory to create the VM. Is it expected that the<br>
> huge pages are<br>
> > "allocated" to each NUMA node?<br>
><br>
> Yes, any given memory page exists on one NUMA node, and a<br>
> single-NUMA-node VM<br>
> will be constrained to a single host NUMA node and will use<br>
> memory from that<br>
> host NUMA node.<br>
><br>
> You can see and/or adjust how many hugepages are available on<br>
> each NUMA node via<br>
> /sys/devices/system/node/nodeX/hugepages/hugepages-2048kB/*<br>
> where X is the host<br>
> NUMA node number.<br>
><br>
> Chris<br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe:<br>
> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div>