Open Stack

Hi Belmiro,


On 06/30/2014 11:42 PM, Belmiro Moreira wrote:
> Hi Eric,
> definitely...
>
> In my view a "FairShareScheduler" could be a very interesting option 
> for private clouds that support scientific communities. Basically this 
> is the model used by batch systems in order to fully use the available 
> resources.
Yes, it is.
> I'm very curious about the work that you are doing.
You can find more info at the following link:
https://agenda.infn.it/getFile.py/access?contribId=17&sessionId=3&resId=0&materialId=slides&confId=7915 

> Is it available in github?
At slide 18, you can find the pointer to the FairShareScheduler.

Cheers,
  Eric.

>
> Belmiro
>
> ----------------------------------
>
> Belmiro Moreira
>
> CERN
>
> Email: belmiro.moreira at cern.ch <mailto:belmiro.moreira at cern.ch>
>
> IRC: belmoreira
>
>
>
> On Mon, Jun 30, 2014 at 4:05 PM, Eric Frizziero 
> <eric.frizziero at pd.infn.it <mailto:eric.frizziero at pd.infn.it>> wrote:
>
>     Hi All,
>
>     we have analyzed the nova-scheduler component (FilterScheduler) in
>     our Openstack installation used by some scientific teams.
>
>     In our scenario, the cloud resources need to be distributed among
>     the teams by considering the predefined share (e.g. quota)
>     assigned to each team, the portion of the resources currently used
>     and the resources they have already consumed.
>
>     We have observed that:
>     1) User requests are sequentially processed (FIFO scheduling),
>     i.e. FilterScheduler doesn't provide any dynamic priority algorithm;
>     2) User requests that cannot be satisfied (e.g. if resources are
>     not available) fail and will be lost, i.e. on that scenario
>     nova-scheduler doesn't provide any queuing of the requests;
>     3) OpenStack simply provides a static partitioning of resources
>     among various projects / teams (use of quotas). If project/team 1
>     in a period is systematically underutilizing its quota and the
>     project/team 2 instead is systematically saturating its quota, the
>     only solution to give more resource to project/team 2 is a manual
>     change (to be done by the admin) to the related quotas.
>
>     The need to find a better approach to enable a more effective
>     scheduling in Openstack becomes more and more evident when the
>     number of the user requests to be handled increases significantly.
>     This is a well known problem which has already been solved in the
>     past for the Batch Systems.
>
>     In order to solve those issues in our usage scenario of Openstack,
>     we have developed a prototype of a pluggable scheduler, named
>     FairShareScheduler, with the objective to extend the existing
>     OpenStack scheduler (FilterScheduler) by integrating a (batch
>     like) dynamic priority algorithm.
>
>     The architecture of the FairShareScheduler is explicitly designed
>     to provide a high scalability level. To all user requests will be
>     assigned a priority value calculated by considering the share
>     allocated to the user by the administrator and the evaluation of
>     the effective resource usage consumed in the recent past. All
>     requests will be inserted in a priority queue, and processed in
>     parallel by a configurable pool of workers without interfering
>     with the priority order. Moreover all significant information
>     (e.g. priority queue) will be stored in a persistence layer in
>     order to provide a fault tolerance mechanism while a proper
>     logging system will annotate all relevant events, useful for
>     auditing processing.
>
>     In more detail, some features of the FairshareScheduler are:
>     a) It assigns dynamically the proper priority to every new user
>     requests;
>     b) The priority of the queued requests will be recalculated
>     periodically using the fairshare algorithm. This feature
>     guarantees the usage of the cloud resources is distributed among
>     users and groups by considering the portion of the cloud resources
>     allocated to them (i.e. share) and the resources already consumed;
>     c) all user requests will be inserted in a (persistent) priority
>     queue and then processed asynchronously by the dedicated process
>     (filtering + weighting phase) when compute resources are available;
>     d) From the client point of view the queued requests remain in
>     "Scheduling" state till the compute resources are available. No
>     new states added: this prevents any possible interaction issue
>     with the Openstack clients;
>     e) User requests are dequeued by a pool of WorkerThreads
>     (configurable), i.e. no sequential processing of the requests;
>     f) The failed requests at filtering + weighting phase may be
>     inserted again in the queue for n-times (configurable).
>
>     We have integrated the FairShareScheduler in our Openstack
>     installation (release "HAVANA"). We're now working to adapt the
>     FairShareScheduler to the new release "IceHouse".
>
>     Does anyone have experiences in those issues found in our cloud
>     scenario?
>
>     Could the FairShareScheduler be useful for the Openstack community?
>     In that case, we'll be happy to share our work.
>
>     Any feedback/comment is welcome!
>
>     Cheers,
>     Eric.
>
>
>     _______________________________________________
>     OpenStack-dev mailing list
>     OpenStack-dev at lists.openstack.org
>     <mailto:OpenStack-dev at lists.openstack.org>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140701/eb894ae9/attachment.html>