Hi Sam & Arne,

This is really helpful. We are actually looking for similar solutions to experiment with. We'll try Aardvark in our lab and let you know if we need any assistance.

We’d also be happy to contribute if the community considers bringing Aardvark back into the upstream.

Regards

Vivek

On Fri, Oct 18, 2024 at 5:36 AM Sam Morrison <sorrison@gmail.com> wrote:

We at Nectar still use Aardvark for preemptible instances. It is a fork of what CERN developed with some modifications. 

How we use is is we have flavours with a property that indicates the flavour is preemptible. Then aardvark will delete these

Instances when capacity is over a threshold or 24 hours, which ever comes first.

We then charge significantly less for these flavours. So the pricing isn’t dynamic like spot instances but it still allows some of our bigger users to chew up some free capacity.

Source: https://github.com/NeCTAR-RC/aardvark

We deploy ours with this helm chart https://github.com/NeCTAR-RC/aardvark-helm

Cheers,

Sam

On 18 Oct 2024, at 2:26 am, Arne Wiebalck <Arne.Wiebalck@cern.ch> wrote:

FWIW, we actually had a full implementation of pre-emptible instances:
it changed name from Reaper to Aardvark and it was used in production
here at CERN until a recent upgrade.

The design and patch sets were discussed with upstream at the time, but
the required change of the state machine (to add the "pending" state)
was the main point which prevented integration into the upstream code
base eventually.

We do not use this functionality in our production deployment at the moment,
mostly since we weighed carrying forward a local patch set versus the benefit
of the added functionality, but the code is of course still there and we're happy
to share it again if there is revived interest in the community.

Cheers,
Arne

--
Arne Wiebalck
CERN IT

________________________________________
From: Julia Kreger <juliaashleykreger@gmail.com>
Sent: Thursday, 17 October 2024 16:23
To: Mia
Cc: openstack-discuss@lists.openstack.org
Subject: Re: Spot Instances

Just an observation following up from discussions I was part of as
well as presentations I listened to at OpenInfra Days NA which
occurred this week. There were a surprising number of mentions of spot
instance-like functionality by some OpenStack vendors and
scientific/academic infrastructure operators.

It might be good for the wider community to revisit such discussion,
given next week is the PTG, as there is clearly renewed interest and
different practices being adopted which creates an inconsistent user
experience across different OpenStack clouds.

-Julia

On Thu, Oct 17, 2024 at 2:14 AM Mia <mikasa.ack1618@gmail.com> wrote:

Thank you both; it was really helpful.

On Wed, Oct 16, 2024 at 11:05 PM Sean Mooney <smooney@redhat.com> wrote:

On 16/10/2024 19:49, Dmitriy Rabotyagov wrote:

Hey,

Somebody may correct me, but to the best of my knowledge this
functionality is currently not implemented in Nova.

it was proposed, discused for a few release and ultimatly rejected.


https://blueprints.launchpad.net/nova/+spec/preemptible-instances
https://review.opendev.org/c/openstack/nova-specs/+/438640
https://review.opendev.org/c/openstack/nova/+/547450


the most recen tspec that was aproved for this was

https://specs.openstack.org/openstack/nova-specs/specs/train/approved/introduce-pending-vm-state.html

and out of tree protype was developed

https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype

but it was never compelted.

after that did not pan out it moved to balzar

https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preemptible-instances.html


and again it never completed to my knowladge

I can assume there could be some custom scheduler filters/weighers in
the wild (not 100% sure this can be done solely through scheduling
though), but I personally haven't seen one so far.
Though I was asked a couple of times during last year about exactly
the same thing. So you're totally not alone looking for a solution for
spot instances.

it is a potential usescase that could be supproted but the logic for
doign the reaping/premetion is non trvial to do correctly.

one approch we have rejected in the past is havign nova's sechuler call
out to another service if we cant find a host

or activly reaping premtiable instnace in that case.

On Wed, Oct 16, 2024, 14:09 Mia <mikasa.ack1618@gmail.com> wrote:

   Hello everyone,

   I am working on a project in the OpenStack environment. More
   precisely, I am looking into the implementation of spot instances
   as a cost optimization strategy and better resource utilization.

   If any community members could share their experiences and
   insights related to the following areas, it would be greatly
   appreciated.
   Implementation Strategies: How did you go about implementing spot
   instances in your cluster?
   Challenges Faced: Mention some of the challenges or limitations.
   Best Practices: Do you have any recommendations or best practices
   that you would like to suggest for seamless deployment?

   Thank you very much for your time and support.