Open Stack

Hi All,

we have analyzed the nova-scheduler component (FilterScheduler) in our 
Openstack installation used by some scientific teams.

In our scenario, the cloud resources need to be distributed among the 
teams by considering the predefined share (e.g. quota) assigned to each 
team, the portion of the resources currently used and the resources they 
have already consumed.

We have observed that:
1) User requests are sequentially processed (FIFO scheduling), i.e. 
FilterScheduler doesn't provide any dynamic priority algorithm;
2) User requests that cannot be satisfied (e.g. if resources are not 
available) fail and will be lost, i.e. on that scenario nova-scheduler 
doesn't provide any queuing of the requests;
3) OpenStack simply provides a static partitioning of resources among 
various projects / teams (use of quotas). If project/team 1 in a period 
is systematically underutilizing its quota and the project/team 2 
instead is systematically saturating its quota, the only solution to 
give more resource to project/team 2 is a manual change (to be done by 
the admin) to the related quotas.

The need to find a better approach to enable a more effective scheduling 
in Openstack becomes more and more evident when the number of the user 
requests to be handled increases significantly. This is a well known 
problem which has already been solved in the past for the Batch Systems.

In order to solve those issues in our usage scenario of Openstack, we 
have developed a prototype of a pluggable scheduler, named 
FairShareScheduler, with the objective to extend the existing OpenStack 
scheduler (FilterScheduler) by integrating a (batch like) dynamic 
priority algorithm.

The architecture of the FairShareScheduler is explicitly designed to 
provide a high scalability level. To all user requests will be assigned 
a priority value calculated by considering the share allocated to the 
user by the administrator and the evaluation of the effective resource 
usage consumed in the recent past. All requests will be inserted in a 
priority queue, and processed in parallel by a configurable pool of 
workers without interfering with the priority order. Moreover all 
significant information (e.g. priority queue) will be stored in a 
persistence layer in order to provide a fault tolerance mechanism while 
a proper logging system will annotate all relevant events, useful for 
auditing processing.

In more detail, some features of the FairshareScheduler are:
a) It assigns dynamically the proper priority to every new user requests;
b) The priority of the queued requests will be recalculated periodically 
using the fairshare algorithm. This feature guarantees the usage of the 
cloud resources is distributed among users and groups by considering the 
portion of the cloud resources allocated to them (i.e. share) and the 
resources already consumed;
c) all user requests will be inserted in a (persistent) priority queue 
and then processed asynchronously by the dedicated process (filtering + 
weighting phase) when compute resources are available;
d) From the client point of view the queued requests remain in 
“Scheduling” state till the compute resources are available. No new 
states added: this prevents any possible interaction issue with the 
Openstack clients;
e) User requests are dequeued by a pool of WorkerThreads (configurable), 
i.e. no sequential processing of the requests;
f) The failed requests at filtering + weighting phase may be inserted 
again in the queue for n-times (configurable).

We have integrated the FairShareScheduler in our Openstack installation 
(release "HAVANA"). We're now working to adapt the FairShareScheduler to 
the new release "IceHouse".

Does anyone have experiences in those issues found in our cloud scenario?

Could the FairShareScheduler be useful for the Openstack community?
In that case, we'll be happy to share our work.

Any feedback/comment is welcome!

Cheers,
Eric.