[Openstack-operators] [scientific] Resource reservation requirements (Blazar) - Forum session
jaypipes at gmail.com
Tue Apr 11 19:22:56 UTC 2017
On 04/11/2017 02:08 PM, Pierre Riteau wrote:
>> On 4 Apr 2017, at 22:23, Jay Pipes <jaypipes at gmail.com
>> <mailto:jaypipes at gmail.com>> wrote:
>> On 04/04/2017 02:48 PM, Tim Bell wrote:
>>> Some combination of spot/OPIE
>> What is OPIE?
> Maybe I missed a message: I didn’t see any reply to Jay’s question about
> OPIE is the OpenStack Preemptible Instances
> Extension: https://github.com/indigo-dc/opie
> I am sure other on this list can provide more information.
> I think running OPIE instances inside Blazar reservations would be
> doable without many changes to the implementation.
> We’ve talked about this idea several times, this forum session would be
> an ideal place to draw up an implementation plan.
I just looked through the OPIE source code. One thing I'm wondering is
why the code for killing off pre-emptible instances is being done in the
Why not have a separate service that merely responds to the raising of a
NoValidHost exception being raised from the scheduler with a call to go
and terminate one or more instances that would have allowed the original
request to land on a host?
Right here is where OPIE goes and terminates pre-emptible instances:
However, that code should actually be run when line 90 raises NoValidHost:
There would be no need at all for "detecting overcommit" here:
Simply detect a NoValidHost being returned to the conductor from the
scheduler, examine if there are pre-emptible instances currently running
that could be terminated and terminate them, and re-run the original
call to select_destinations() (the scheduler call) just like a Retry
operation normally does.
There's be no need whatsoever to involve any changes to the scheduler at
>>> and Blazar would seem doable as long as the resource provider
>>> reserves capacity appropriately (i.e. spot resources>>blazar
>>> committed along with no non-spot requests for the same aggregate).
>>> Is this feasible?
As mentioned in previous emails and on the etherpad here:
I am firmly against having the resource tracker or the placement API
represent inventory or allocations with a temporal aspect to them (i.e.
allocations in the future).
A separate system (hopefully Blazar) is needed to manage the time-based
associations to inventories of resources over a period in the future.
>> I'm not sure how the above is different from the constraints I mention
>> below about having separate sets of resource providers for preemptible
>> instances than for non-preemptible instances?
>>> On 04.04.17, 19:21, "Jay Pipes" <jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>> wrote:
>>> On 04/03/2017 06:07 PM, Blair Bethwaite wrote:
>>> > Hi Jay,
>>> > On 4 April 2017 at 00:20, Jay Pipes <jaypipes at gmail.com
>>> <mailto:jaypipes at gmail.com>> wrote:
>>> >> However, implementing the above in any useful fashion requires
>>> that Blazar
>>> >> be placed *above* Nova and essentially that the cloud operator
>>> turns off
>>> >> access to Nova's POST /servers API call for regular users.
>>> Because if not,
>>> >> the information that Blazar acts upon can be simply
>>> circumvented by any user
>>> >> at any time.
>>> > That's something of an oversimplification. A reservation system
>>> > outside of Nova could manipulate Nova host-aggregates to "cordon
>>> > infrastructure from on-demand access (I believe Blazar already uses
>>> > this approach), and it's not much of a jump to imagine operators
>>> > able to twiddle the available reserved capacity in a finite cloud so
>>> > that reserved capacity can be offered to the subset of
>>> > that need (or perhaps have paid for) it.
>>> Sure, I'm following you up until here.
>>> > Such a reservation system would even be able to backfill capacity
>>> > between reservations. At the end of the reservation the system
>>> > cleans-up any remaining instances and preps for the next
>>> > reservation.
>>> By "backfill capacity between reservations", do you mean consume
>>> resources on the compute hosts that are "reserved" by this paying
>>> customer at some date in the future? i.e. Spot instances that can be
>>> killed off as necessary by the reservation system to free resources to
>>> meet its reservation schedule?
>>> > The are a couple of problems with putting this outside of Nova
>>> > The main issue is that pre-emptible/spot type instances can't be
>>> > accommodated within the on-demand cloud capacity.
>>> Correct. The reservation system needs complete control over a
>>> subset of
>>> resource providers to be used for these spot instances. It would
>>> be like
>>> a hotel reservation system being used for a motel where cars could
>>> simply pull up to a room with a vacant sign outside the door. The
>>> reservation system would never be able to work on accurate data unless
>>> some part of the motel's rooms were carved out for reservation
>>> system to
>>> use and cars to not pull up and take.
>>> > You could have the
>>> > reservation system implementing this feature, but that would
>>> then put
>>> > other scheduling constraints on the cloud in order to be effective
>>> > (e.g., there would need to be automation changing the size of the
>>> > on-demand capacity so that the maximum pre-emptible capacity was
>>> > always available). The other issue (admittedly minor, but still a
>>> > consideration) is that it's another service - personally I'd love to
>>> > see Nova support these advanced use-cases directly.
>>> Welcome to the world of microservices. :)
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> <mailto:OpenStack-operators at lists.openstack.org>
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> <mailto:OpenStack-operators at lists.openstack.org>
More information about the OpenStack-operators