<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div><span style="font-size: 12pt;">Well, I don't know what John means by "modify the </span>over-commit calculation in the scheduler", so I cannot comment. </div><div><br></div><div><span style="font-size: 12pt;">The idea of choosing free host for Hadoop on the fly is rather complicated and contains several operations, namely: (1) assuring the host never get pass 100% CPU load; (2) identifying a host that already has a Hadoop VM running on it, or already 100% CPU commitment; (3) releasing the host from 100% CPU commitment once the Hadoop VM stops; (4) possibly avoiding other applications to use the host (to economy the host resource).</span></div><div><div><span style="font-size: 12pt;"><br></span></div><div><span style="font-size: 12pt;">- You'll need (1) because otherwise your Hadoop VM would come short of resources after the host gets overloaded.</span></div><div><span style="font-size: 12pt;">- You'll need (2) because you don't want to restrict a new host while one of your 100% CPU commited hosts still has free resources.</span></div><div><span style="font-size: 12pt;">- You'll need (3) because otherwise you host would be forerever restricted, and that is no longer "on the fly".</span></div><div><span style="font-size: 12pt;">- You'll may need (4) because otherwise it'd be a waste of resources.</span></div><div><span style="font-size: 12pt;"><br></span></div><div><span style="font-size: 12pt;">The problem of changing CPU overcommit on the fly is that when your Hadoop VM is still running, someone else can add another VM in the same host with a higher CPU overcommit (e.g. 200%), (violating (1) ) thus effecting your Hadoop VM also. </span></div></div><div><span style="font-size: 12pt;">The idea of putting the host in the aggregate can give you (1) and (2). (4) is done by </span>AggregateInstanceExtraSpecsFilter. However, it does not give you (3); which can be done with pCloud.</div><div><br></div><div><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Alexander Kuznetsov" <akuznetsov@mirantis.com><br><b>To: </b>"OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org><br><b>Sent: </b>Wednesday, November 13, 2013 3:09:40 PM<br><b>Subject: </b>Re: [openstack-dev] [nova] Configure overcommit policy<br><div><br></div><div dir="ltr">Toan and Alex. Having separate computes pools for Hadoop is not suitable if we want to use an unused power of OpenStack cluster to run Hadoop analytic jobs. Possibly in this case it is better to modify the over-commit calculation in the scheduler according John suggestion.<br>
</div><div class="gmail_extra"><br><div><br></div><div class="gmail_quote">On Tue, Nov 12, 2013 at 7:16 PM, Khanh-Toan Tran <span dir="ltr"><<a href="mailto:khanh-toan.tran@cloudwatt.com" target="_blank">khanh-toan.tran@cloudwatt.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:times new roman,new york,times,serif"><div><span style="font-family:arial,helvetica,sans-serif">FYI, by default Openstack overcommit CPU 1:16, meaning it can host 16 times number of cores it possesses. As mentioned Alex, you can change it by enabling AggregateCoreFilter in nova.conf:</span></div>
<div><span style="font-family:arial,helvetica,sans-serif"> scheduler_default_filters = <list of your filters, adding AggregateCoreFilter here></span><br></div><div><span style="font-family:arial,helvetica,sans-serif"><br>
</span></div><div><span style="font-family:arial,helvetica,sans-serif">and modifying the overcommit ratio by adding:</span></div><div><span style="font-family:arial,helvetica,sans-serif"> cpu_allocation_ratio=1.0</span></div>
<div><span style="font-family:arial,helvetica,sans-serif"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif">Just a suggestion, think of isolating the host for the tenant that uses Hadoop so that it will not serve other applications. You have several filters at your disposal:</span></div>
<div><span style="font-family:arial,helvetica,sans-serif"> AggregateInstanceExtraSpecsFilter</span></div><div><span style="font-family:arial,helvetica,sans-serif"> IsolatedHostsFilter</span></div><div><span style="font-family:arial,helvetica,sans-serif"> AggregateMultiTenancyIsolation</span></div>
<div><span style="font-family:arial,helvetica,sans-serif"><br></span></div><div><span style="font-family:arial,helvetica,sans-serif">Best regards,</span></div><div><span style="font-family:arial,helvetica,sans-serif"><br>
</span></div><div><span style="font-family:arial,helvetica,sans-serif">Toan</span></div><div><span style="font-family:Verdana,Geneva,sans-serif;font-size:13px"><br></span></div><hr><div style="font-size:12pt;font-style:normal;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal">
<b>From: </b>"Alex Glikson" <<a href="mailto:GLIKSON@il.ibm.com" target="_blank">GLIKSON@il.ibm.com</a>><div class="im"><br><b>To: </b>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
</div><b>Sent: </b>Tuesday, November 12, 2013 3:54:02 PM<div><div class="h5"><br><b>Subject: </b>Re: [openstack-dev] [nova] Configure overcommit policy<br><div><br></div><span style="font-family:sans-serif;font-size:small">You can consider having a separate host
aggregate for Hadoop, and use a combination of AggregateInstanceExtraSpecFilter
(with a special flavor mapped to this host aggregate) and AggregateCoreFilter
(overriding cpu_allocation_ratio for this host aggregate to be 1).</span>
<br>
<br><span style="font-family:sans-serif;font-size:small">Regards,</span>
<br><span style="font-family:sans-serif;font-size:small">Alex</span>
<br>
<br>
<br>
<br>
<br><span style="color:#5f5f5f;font-family:sans-serif;font-size:xx-small">From:
</span><span style="font-family:sans-serif;font-size:xx-small">John Garbutt <<a href="mailto:john@johngarbutt.com" target="_blank">john@johngarbutt.com</a>></span>
<br><span style="color:#5f5f5f;font-family:sans-serif;font-size:xx-small">To:
</span><span style="font-family:sans-serif;font-size:xx-small">"OpenStack Development
Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>>,
</span>
<br><span style="color:#5f5f5f;font-family:sans-serif;font-size:xx-small">Date:
</span><span style="font-family:sans-serif;font-size:xx-small">12/11/2013 04:41 PM</span>
<br><span style="color:#5f5f5f;font-family:sans-serif;font-size:xx-small">Subject:
</span><span style="font-family:sans-serif;font-size:xx-small">Re: [openstack-dev]
[nova] Configure overcommit policy</span>
<br>
<hr noshade="">
<br>
<br>
<br><tt><span style="font-size:small">On 11 November 2013 12:04, Alexander Kuznetsov <<a href="mailto:akuznetsov@mirantis.com" target="_blank">akuznetsov@mirantis.com</a>>
wrote:<br>
> Hi all,<br>
><br>
> While studying Hadoop performance in a virtual environment, I found
an<br>
> interesting problem with Nova scheduling. In OpenStack cluster, we
have<br>
> overcommit policy, allowing to put on one compute more vms than resources<br>
> available for them. While it might be suitable for general types of<br>
> workload, this is definitely not the case for Hadoop clusters, which
usually<br>
> consume 100% of system resources.<br>
><br>
> Is there any way to tell Nova to schedule specific instances (the
ones which<br>
> consume 100% of system resources) without overcommitting resources
on<br>
> compute node?<br>
<br>
You could have a flavor with "no-overcommit" extra spec, and
modify<br>
the over-commit calculation in the scheduler on that case, but I don't<br>
remember seeing that in there.<br>
<br>
John<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
</span></tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><tt><span style="font-size:small">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></tt></a><tt><span style="font-size:small"><br>
<br>
</span></tt>
<br><div><br></div>_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></div><div><br></div><div style="width:1px;min-height:1px;overflow:hidden"></div></div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>OpenStack-dev mailing list<br>OpenStack-dev@lists.openstack.org<br>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev<br></div><div><br></div><div id="_mcePaste" class="mcePaste" data-mce-bogus="1" style="position: absolute; left: 0px; top: -25px; width: 1px; height: 1px; overflow: hidden;"></div></div></body></html>