Open Stack

Toan and Alex. Having separate computes pools for Hadoop is not suitable if
we want to use  an unused power of OpenStack cluster to run Hadoop analytic
 jobs. Possibly in this case it is better to modify the over-commit
calculation in the scheduler according John suggestion.


On Tue, Nov 12, 2013 at 7:16 PM, Khanh-Toan Tran <
khanh-toan.tran at cloudwatt.com> wrote:

> FYI, by default Openstack overcommit CPU 1:16, meaning it can host 16
> times number of cores it possesses. As mentioned Alex, you can change it by
> enabling AggregateCoreFilter in nova.conf:
>    scheduler_default_filters = <list of your filters,
> adding AggregateCoreFilter here>
>
> and modifying the overcommit ratio by adding:
>   cpu_allocation_ratio=1.0
>
> Just a suggestion, think of isolating the host for the tenant that uses
> Hadoop so that it will not serve other applications. You have several
> filters at your disposal:
>      AggregateInstanceExtraSpecsFilter
>      IsolatedHostsFilter
>      AggregateMultiTenancyIsolation
>
> Best regards,
>
> Toan
>
> ------------------------------
> *From: *"Alex Glikson" <GLIKSON at il.ibm.com>
>
> *To: *"OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev at lists.openstack.org>
> *Sent: *Tuesday, November 12, 2013 3:54:02 PM
>
> *Subject: *Re: [openstack-dev] [nova] Configure overcommit policy
>
> You can consider having a separate host aggregate for Hadoop, and use a
> combination of AggregateInstanceExtraSpecFilter (with a special flavor
> mapped to this host aggregate) and AggregateCoreFilter (overriding
> cpu_allocation_ratio for this host aggregate to be 1).
>
> Regards,
> Alex
>
>
>
>
> From:        John Garbutt <john at johngarbutt.com>
> To:        "OpenStack Development Mailing List (not for usage questions)"
> <openstack-dev at lists.openstack.org>,
> Date:        12/11/2013 04:41 PM
> Subject:        Re: [openstack-dev] [nova] Configure overcommit policy
> ------------------------------
>
>
>
> On 11 November 2013 12:04, Alexander Kuznetsov <akuznetsov at mirantis.com>
> wrote:
> > Hi all,
> >
> > While studying Hadoop performance in a virtual environment, I found an
> > interesting problem with Nova scheduling. In OpenStack cluster, we have
> > overcommit policy, allowing to put on one compute more vms than resources
> > available for them. While it might be suitable for general types of
> > workload, this is definitely not the case for Hadoop clusters, which
> usually
> > consume 100% of system resources.
> >
> > Is there any way to tell Nova to schedule specific instances (the ones
> which
> > consume 100% of system resources) without overcommitting resources on
> > compute node?
>
> You could have a flavor with "no-overcommit" extra spec, and modify
> the over-commit calculation in the scheduler on that case, but I don't
> remember seeing that in there.
>
> John
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> 
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20131113/48c6d57d/attachment.html>