Hello everyone, I am working on a project in the OpenStack environment. More precisely, I am looking into the implementation of spot instances as a cost optimization strategy and better resource utilization. If any community members could share their experiences and insights related to the following areas, it would be greatly appreciated. Implementation Strategies: How did you go about implementing spot instances in your cluster? Challenges Faced: Mention some of the challenges or limitations. Best Practices: Do you have any recommendations or best practices that you would like to suggest for seamless deployment? Thank you very much for your time and support.
Hey, Somebody may correct me, but to the best of my knowledge this functionality is currently not implemented in Nova. I can assume there could be some custom scheduler filters/weighers in the wild (not 100% sure this can be done solely through scheduling though), but I personally haven't seen one so far. Though I was asked a couple of times during last year about exactly the same thing. So you're totally not alone looking for a solution for spot instances. On Wed, Oct 16, 2024, 14:09 Mia <mikasa.ack1618@gmail.com> wrote:
Hello everyone,
I am working on a project in the OpenStack environment. More precisely, I am looking into the implementation of spot instances as a cost optimization strategy and better resource utilization.
If any community members could share their experiences and insights related to the following areas, it would be greatly appreciated. Implementation Strategies: How did you go about implementing spot instances in your cluster? Challenges Faced: Mention some of the challenges or limitations. Best Practices: Do you have any recommendations or best practices that you would like to suggest for seamless deployment?
Thank you very much for your time and support.
On 16/10/2024 19:49, Dmitriy Rabotyagov wrote:
Hey,
Somebody may correct me, but to the best of my knowledge this functionality is currently not implemented in Nova.
it was proposed, discused for a few release and ultimatly rejected. https://blueprints.launchpad.net/nova/+spec/preemptible-instances https://review.opendev.org/c/openstack/nova-specs/+/438640 https://review.opendev.org/c/openstack/nova/+/547450 the most recen tspec that was aproved for this was https://specs.openstack.org/openstack/nova-specs/specs/train/approved/introd... and out of tree protype was developed https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype but it was never compelted. after that did not pan out it moved to balzar https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem... and again it never completed to my knowladge
I can assume there could be some custom scheduler filters/weighers in the wild (not 100% sure this can be done solely through scheduling though), but I personally haven't seen one so far. Though I was asked a couple of times during last year about exactly the same thing. So you're totally not alone looking for a solution for spot instances.
it is a potential usescase that could be supproted but the logic for doign the reaping/premetion is non trvial to do correctly. one approch we have rejected in the past is havign nova's sechuler call out to another service if we cant find a host or activly reaping premtiable instnace in that case.
On Wed, Oct 16, 2024, 14:09 Mia <mikasa.ack1618@gmail.com> wrote:
Hello everyone,
I am working on a project in the OpenStack environment. More precisely, I am looking into the implementation of spot instances as a cost optimization strategy and better resource utilization.
If any community members could share their experiences and insights related to the following areas, it would be greatly appreciated. Implementation Strategies: How did you go about implementing spot instances in your cluster? Challenges Faced: Mention some of the challenges or limitations. Best Practices: Do you have any recommendations or best practices that you would like to suggest for seamless deployment?
Thank you very much for your time and support.
Thank you both; it was really helpful. On Wed, Oct 16, 2024 at 11:05 PM Sean Mooney <smooney@redhat.com> wrote:
On 16/10/2024 19:49, Dmitriy Rabotyagov wrote:
Hey,
Somebody may correct me, but to the best of my knowledge this functionality is currently not implemented in Nova.
it was proposed, discused for a few release and ultimatly rejected.
https://blueprints.launchpad.net/nova/+spec/preemptible-instances https://review.opendev.org/c/openstack/nova-specs/+/438640 https://review.opendev.org/c/openstack/nova/+/547450
the most recen tspec that was aproved for this was
https://specs.openstack.org/openstack/nova-specs/specs/train/approved/introd...
and out of tree protype was developed
https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype
but it was never compelted.
after that did not pan out it moved to balzar
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
and again it never completed to my knowladge
I can assume there could be some custom scheduler filters/weighers in the wild (not 100% sure this can be done solely through scheduling though), but I personally haven't seen one so far. Though I was asked a couple of times during last year about exactly the same thing. So you're totally not alone looking for a solution for spot instances.
it is a potential usescase that could be supproted but the logic for doign the reaping/premetion is non trvial to do correctly.
one approch we have rejected in the past is havign nova's sechuler call out to another service if we cant find a host
or activly reaping premtiable instnace in that case.
On Wed, Oct 16, 2024, 14:09 Mia <mikasa.ack1618@gmail.com> wrote:
Hello everyone,
I am working on a project in the OpenStack environment. More precisely, I am looking into the implementation of spot instances as a cost optimization strategy and better resource utilization.
If any community members could share their experiences and insights related to the following areas, it would be greatly appreciated. Implementation Strategies: How did you go about implementing spot instances in your cluster? Challenges Faced: Mention some of the challenges or limitations. Best Practices: Do you have any recommendations or best practices that you would like to suggest for seamless deployment?
Thank you very much for your time and support.
Just an observation following up from discussions I was part of as well as presentations I listened to at OpenInfra Days NA which occurred this week. There were a surprising number of mentions of spot instance-like functionality by some OpenStack vendors and scientific/academic infrastructure operators. It might be good for the wider community to revisit such discussion, given next week is the PTG, as there is clearly renewed interest and different practices being adopted which creates an inconsistent user experience across different OpenStack clouds. -Julia On Thu, Oct 17, 2024 at 2:14 AM Mia <mikasa.ack1618@gmail.com> wrote:
Thank you both; it was really helpful.
On Wed, Oct 16, 2024 at 11:05 PM Sean Mooney <smooney@redhat.com> wrote:
On 16/10/2024 19:49, Dmitriy Rabotyagov wrote:
Hey,
Somebody may correct me, but to the best of my knowledge this functionality is currently not implemented in Nova.
it was proposed, discused for a few release and ultimatly rejected.
https://blueprints.launchpad.net/nova/+spec/preemptible-instances https://review.opendev.org/c/openstack/nova-specs/+/438640 https://review.opendev.org/c/openstack/nova/+/547450
the most recen tspec that was aproved for this was
https://specs.openstack.org/openstack/nova-specs/specs/train/approved/introd...
and out of tree protype was developed
https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype
but it was never compelted.
after that did not pan out it moved to balzar
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
and again it never completed to my knowladge
I can assume there could be some custom scheduler filters/weighers in the wild (not 100% sure this can be done solely through scheduling though), but I personally haven't seen one so far. Though I was asked a couple of times during last year about exactly the same thing. So you're totally not alone looking for a solution for spot instances.
it is a potential usescase that could be supproted but the logic for doign the reaping/premetion is non trvial to do correctly.
one approch we have rejected in the past is havign nova's sechuler call out to another service if we cant find a host
or activly reaping premtiable instnace in that case.
On Wed, Oct 16, 2024, 14:09 Mia <mikasa.ack1618@gmail.com> wrote:
Hello everyone,
I am working on a project in the OpenStack environment. More precisely, I am looking into the implementation of spot instances as a cost optimization strategy and better resource utilization.
If any community members could share their experiences and insights related to the following areas, it would be greatly appreciated. Implementation Strategies: How did you go about implementing spot instances in your cluster? Challenges Faced: Mention some of the challenges or limitations. Best Practices: Do you have any recommendations or best practices that you would like to suggest for seamless deployment?
Thank you very much for your time and support.
FWIW, we actually had a full implementation of pre-emptible instances: it changed name from Reaper to Aardvark and it was used in production here at CERN until a recent upgrade. The design and patch sets were discussed with upstream at the time, but the required change of the state machine (to add the "pending" state) was the main point which prevented integration into the upstream code base eventually. We do not use this functionality in our production deployment at the moment, mostly since we weighed carrying forward a local patch set versus the benefit of the added functionality, but the code is of course still there and we're happy to share it again if there is revived interest in the community. Cheers, Arne -- Arne Wiebalck CERN IT ________________________________________ From: Julia Kreger <juliaashleykreger@gmail.com> Sent: Thursday, 17 October 2024 16:23 To: Mia Cc: openstack-discuss@lists.openstack.org Subject: Re: Spot Instances Just an observation following up from discussions I was part of as well as presentations I listened to at OpenInfra Days NA which occurred this week. There were a surprising number of mentions of spot instance-like functionality by some OpenStack vendors and scientific/academic infrastructure operators. It might be good for the wider community to revisit such discussion, given next week is the PTG, as there is clearly renewed interest and different practices being adopted which creates an inconsistent user experience across different OpenStack clouds. -Julia On Thu, Oct 17, 2024 at 2:14 AM Mia <mikasa.ack1618@gmail.com> wrote:
Thank you both; it was really helpful.
On Wed, Oct 16, 2024 at 11:05 PM Sean Mooney <smooney@redhat.com> wrote:
On 16/10/2024 19:49, Dmitriy Rabotyagov wrote:
Hey,
Somebody may correct me, but to the best of my knowledge this functionality is currently not implemented in Nova.
it was proposed, discused for a few release and ultimatly rejected.
https://blueprints.launchpad.net/nova/+spec/preemptible-instances https://review.opendev.org/c/openstack/nova-specs/+/438640 https://review.opendev.org/c/openstack/nova/+/547450
the most recen tspec that was aproved for this was
https://specs.openstack.org/openstack/nova-specs/specs/train/approved/introd...
and out of tree protype was developed
https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype
but it was never compelted.
after that did not pan out it moved to balzar
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
and again it never completed to my knowladge
I can assume there could be some custom scheduler filters/weighers in the wild (not 100% sure this can be done solely through scheduling though), but I personally haven't seen one so far. Though I was asked a couple of times during last year about exactly the same thing. So you're totally not alone looking for a solution for spot instances.
it is a potential usescase that could be supproted but the logic for doign the reaping/premetion is non trvial to do correctly.
one approch we have rejected in the past is havign nova's sechuler call out to another service if we cant find a host
or activly reaping premtiable instnace in that case.
On Wed, Oct 16, 2024, 14:09 Mia <mikasa.ack1618@gmail.com> wrote:
Hello everyone,
I am working on a project in the OpenStack environment. More precisely, I am looking into the implementation of spot instances as a cost optimization strategy and better resource utilization.
If any community members could share their experiences and insights related to the following areas, it would be greatly appreciated. Implementation Strategies: How did you go about implementing spot instances in your cluster? Challenges Faced: Mention some of the challenges or limitations. Best Practices: Do you have any recommendations or best practices that you would like to suggest for seamless deployment?
Thank you very much for your time and support.
Hi Arne. I’d be very interesting in taking on where you left off with Spot instances. Could CERN please share the codebase so I can look at it? Thanks, Karl. Karl Kloppenborg Chief Technology Officer m: +61 437 239 565 resetdata.com<https://resetdata.com/> [cid:reset_69557fc2-1d63-4932-b5fd-93bd4f39ca7b.png] ResetData supports Mandatory Client Related Financial Disclosures – Scope 3 Emissions Reporting For more information on the phasing of these requirements for business please visit; https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-state.pdf<https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-state.pdf> This email transmission is intended only for the addressee / person responsible for delivery of the message to such person and may contain confidential or privileged information. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you, nor may you use, review, disclose, disseminate or copy any information contained in or attached to it. Whilst this email has been checked for viruses, the sender does not warrant that any attachments are free from viruses or other defects. You assume all liability for any loss, damage or other consequences which may arise from opening or using the attachments. If you received this e-mail in error please delete it and any attachments and kindly notify us by immediately sending an email to contact@resetdata.com.au<mailto:contact@resetdata.com.au> From: Arne Wiebalck <Arne.Wiebalck@cern.ch> Date: Friday, 18 October 2024 at 2:27 am To: Julia Kreger <juliaashleykreger@gmail.com>, Mia <mikasa.ack1618@gmail.com> Cc: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> Subject: Re: Spot Instances [You don't often get email from arne.wiebalck@cern.ch. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] FWIW, we actually had a full implementation of pre-emptible instances: it changed name from Reaper to Aardvark and it was used in production here at CERN until a recent upgrade. The design and patch sets were discussed with upstream at the time, but the required change of the state machine (to add the "pending" state) was the main point which prevented integration into the upstream code base eventually. We do not use this functionality in our production deployment at the moment, mostly since we weighed carrying forward a local patch set versus the benefit of the added functionality, but the code is of course still there and we're happy to share it again if there is revived interest in the community. Cheers, Arne -- Arne Wiebalck CERN IT ________________________________________ From: Julia Kreger <juliaashleykreger@gmail.com> Sent: Thursday, 17 October 2024 16:23 To: Mia Cc: openstack-discuss@lists.openstack.org Subject: Re: Spot Instances Just an observation following up from discussions I was part of as well as presentations I listened to at OpenInfra Days NA which occurred this week. There were a surprising number of mentions of spot instance-like functionality by some OpenStack vendors and scientific/academic infrastructure operators. It might be good for the wider community to revisit such discussion, given next week is the PTG, as there is clearly renewed interest and different practices being adopted which creates an inconsistent user experience across different OpenStack clouds. -Julia On Thu, Oct 17, 2024 at 2:14 AM Mia <mikasa.ack1618@gmail.com> wrote:
Thank you both; it was really helpful.
On Wed, Oct 16, 2024 at 11:05 PM Sean Mooney <smooney@redhat.com> wrote:
On 16/10/2024 19:49, Dmitriy Rabotyagov wrote:
Hey,
Somebody may correct me, but to the best of my knowledge this functionality is currently not implemented in Nova.
it was proposed, discused for a few release and ultimatly rejected.
https://blueprints.launchpad.net/nova/+spec/preemptible-instances https://review.opendev.org/c/openstack/nova-specs/+/438640 https://review.opendev.org/c/openstack/nova/+/547450
the most recen tspec that was aproved for this was
https://specs.openstack.org/openstack/nova-specs/specs/train/approved/introd...
and out of tree protype was developed
https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype
but it was never compelted.
after that did not pan out it moved to balzar
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
and again it never completed to my knowladge
I can assume there could be some custom scheduler filters/weighers in the wild (not 100% sure this can be done solely through scheduling though), but I personally haven't seen one so far. Though I was asked a couple of times during last year about exactly the same thing. So you're totally not alone looking for a solution for spot instances.
it is a potential usescase that could be supproted but the logic for doign the reaping/premetion is non trvial to do correctly.
one approch we have rejected in the past is havign nova's sechuler call out to another service if we cant find a host
or activly reaping premtiable instnace in that case.
On Wed, Oct 16, 2024, 14:09 Mia <mikasa.ack1618@gmail.com> wrote:
Hello everyone,
I am working on a project in the OpenStack environment. More precisely, I am looking into the implementation of spot instances as a cost optimization strategy and better resource utilization.
If any community members could share their experiences and insights related to the following areas, it would be greatly appreciated. Implementation Strategies: How did you go about implementing spot instances in your cluster? Challenges Faced: Mention some of the challenges or limitations. Best Practices: Do you have any recommendations or best practices that you would like to suggest for seamless deployment?
Thank you very much for your time and support.
Karl, all, https://gitlab.cern.ch/cloud-infrastructure/aardvark contains the code. There is also a blog post we wrote about it: https://techblog.web.cern.ch/techblog/post/preemptible-instances/ Cheers, Arne -- Arne Wiebalck CERN IT ________________________________________ From: Karl Kloppenborg <kkloppenborg@resetdata.com.au> Sent: Friday, 18 October 2024 01:22 To: Arne Wiebalck; Julia Kreger; Mia Cc: openstack-discuss@lists.openstack.org Subject: Re: Spot Instances Hi Arne. I’d be very interesting in taking on where you left off with Spot instances. Could CERN please share the codebase so I can look at it? Thanks, Karl. Karl Kloppenborg Chief Technology Officer m: +61 437 239 565 resetdata.com<https://resetdata.com/> [reset.png] ResetData supports Mandatory Client Related Financial Disclosures – Scope 3 Emissions Reporting For more information on the phasing of these requirements for business please visit; https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-stat... This email transmission is intended only for the addressee / person responsible for delivery of the message to such person and may contain confidential or privileged information. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you, nor may you use, review, disclose, disseminate or copy any information contained in or attached to it. Whilst this email has been checked for viruses, the sender does not warrant that any attachments are free from viruses or other defects. You assume all liability for any loss, damage or other consequences which may arise from opening or using the attachments. If you received this e-mail in error please delete it and any attachments and kindly notify us by immediately sending an email to contact@resetdata.com.au<mailto:contact@resetdata.com.au> From: Arne Wiebalck <Arne.Wiebalck@cern.ch> Date: Friday, 18 October 2024 at 2:27 am To: Julia Kreger <juliaashleykreger@gmail.com>, Mia <mikasa.ack1618@gmail.com> Cc: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org> Subject: Re: Spot Instances [You don't often get email from arne.wiebalck@cern.ch. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ] FWIW, we actually had a full implementation of pre-emptible instances: it changed name from Reaper to Aardvark and it was used in production here at CERN until a recent upgrade. The design and patch sets were discussed with upstream at the time, but the required change of the state machine (to add the "pending" state) was the main point which prevented integration into the upstream code base eventually. We do not use this functionality in our production deployment at the moment, mostly since we weighed carrying forward a local patch set versus the benefit of the added functionality, but the code is of course still there and we're happy to share it again if there is revived interest in the community. Cheers, Arne -- Arne Wiebalck CERN IT ________________________________________ From: Julia Kreger <juliaashleykreger@gmail.com> Sent: Thursday, 17 October 2024 16:23 To: Mia Cc: openstack-discuss@lists.openstack.org Subject: Re: Spot Instances Just an observation following up from discussions I was part of as well as presentations I listened to at OpenInfra Days NA which occurred this week. There were a surprising number of mentions of spot instance-like functionality by some OpenStack vendors and scientific/academic infrastructure operators. It might be good for the wider community to revisit such discussion, given next week is the PTG, as there is clearly renewed interest and different practices being adopted which creates an inconsistent user experience across different OpenStack clouds. -Julia On Thu, Oct 17, 2024 at 2:14 AM Mia <mikasa.ack1618@gmail.com> wrote:
Thank you both; it was really helpful.
On Wed, Oct 16, 2024 at 11:05 PM Sean Mooney <smooney@redhat.com> wrote:
On 16/10/2024 19:49, Dmitriy Rabotyagov wrote:
Hey,
Somebody may correct me, but to the best of my knowledge this functionality is currently not implemented in Nova.
it was proposed, discused for a few release and ultimatly rejected.
https://blueprints.launchpad.net/nova/+spec/preemptible-instances https://review.opendev.org/c/openstack/nova-specs/+/438640 https://review.opendev.org/c/openstack/nova/+/547450
the most recen tspec that was aproved for this was
https://specs.openstack.org/openstack/nova-specs/specs/train/approved/introd...
and out of tree protype was developed
https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype
but it was never compelted.
after that did not pan out it moved to balzar
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
and again it never completed to my knowladge
I can assume there could be some custom scheduler filters/weighers in the wild (not 100% sure this can be done solely through scheduling though), but I personally haven't seen one so far. Though I was asked a couple of times during last year about exactly the same thing. So you're totally not alone looking for a solution for spot instances.
it is a potential usescase that could be supproted but the logic for doign the reaping/premetion is non trvial to do correctly.
one approch we have rejected in the past is havign nova's sechuler call out to another service if we cant find a host
or activly reaping premtiable instnace in that case.
On Wed, Oct 16, 2024, 14:09 Mia <mikasa.ack1618@gmail.com> wrote:
Hello everyone,
I am working on a project in the OpenStack environment. More precisely, I am looking into the implementation of spot instances as a cost optimization strategy and better resource utilization.
If any community members could share their experiences and insights related to the following areas, it would be greatly appreciated. Implementation Strategies: How did you go about implementing spot instances in your cluster? Challenges Faced: Mention some of the challenges or limitations. Best Practices: Do you have any recommendations or best practices that you would like to suggest for seamless deployment?
Thank you very much for your time and support.
Hi Arne, Thank you for sharing these links. It is very helpful. Will keep you posted as we make progress. Regards Vivek On Mon, Oct 21, 2024 at 12:52 PM Arne Wiebalck <Arne.Wiebalck@cern.ch> wrote:
Karl, all,
https://gitlab.cern.ch/cloud-infrastructure/aardvark contains the code.
There is also a blog post we wrote about it: https://techblog.web.cern.ch/techblog/post/preemptible-instances/
Cheers, Arne
-- Arne Wiebalck CERN IT
________________________________________ From: Karl Kloppenborg <kkloppenborg@resetdata.com.au> Sent: Friday, 18 October 2024 01:22 To: Arne Wiebalck; Julia Kreger; Mia Cc: openstack-discuss@lists.openstack.org Subject: Re: Spot Instances
Hi Arne.
I’d be very interesting in taking on where you left off with Spot instances.
Could CERN please share the codebase so I can look at it?
Thanks, Karl.
Karl Kloppenborg
Chief Technology Officer
m: +61 437 239 565 resetdata.com<https://resetdata.com/>
[reset.png]
ResetData supports Mandatory Client Related Financial Disclosures – Scope 3 Emissions Reporting For more information on the phasing of these requirements for business please visit;
https://treasury.gov.au/sites/default/files/2024-01/c2024-466491-policy-stat...
This email transmission is intended only for the addressee / person responsible for delivery of the message to such person and may contain confidential or privileged information. Confidentiality and legal privilege are not waived or lost by reason of mistaken delivery to you, nor may you use, review, disclose, disseminate or copy any information contained in or attached to it. Whilst this email has been checked for viruses, the sender does not warrant that any attachments are free from viruses or other defects. You assume all liability for any loss, damage or other consequences which may arise from opening or using the attachments. If you received this e-mail in error please delete it and any attachments and kindly notify us by immediately sending an email to contact@resetdata.com.au<mailto:contact@resetdata.com.au> From: Arne Wiebalck <Arne.Wiebalck@cern.ch> Date: Friday, 18 October 2024 at 2:27 am To: Julia Kreger <juliaashleykreger@gmail.com>, Mia < mikasa.ack1618@gmail.com> Cc: openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> Subject: Re: Spot Instances [You don't often get email from arne.wiebalck@cern.ch. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
FWIW, we actually had a full implementation of pre-emptible instances: it changed name from Reaper to Aardvark and it was used in production here at CERN until a recent upgrade.
The design and patch sets were discussed with upstream at the time, but the required change of the state machine (to add the "pending" state) was the main point which prevented integration into the upstream code base eventually.
We do not use this functionality in our production deployment at the moment, mostly since we weighed carrying forward a local patch set versus the benefit of the added functionality, but the code is of course still there and we're happy to share it again if there is revived interest in the community.
Cheers, Arne
-- Arne Wiebalck CERN IT
________________________________________ From: Julia Kreger <juliaashleykreger@gmail.com> Sent: Thursday, 17 October 2024 16:23 To: Mia Cc: openstack-discuss@lists.openstack.org Subject: Re: Spot Instances
Just an observation following up from discussions I was part of as well as presentations I listened to at OpenInfra Days NA which occurred this week. There were a surprising number of mentions of spot instance-like functionality by some OpenStack vendors and scientific/academic infrastructure operators.
It might be good for the wider community to revisit such discussion, given next week is the PTG, as there is clearly renewed interest and different practices being adopted which creates an inconsistent user experience across different OpenStack clouds.
-Julia
On Thu, Oct 17, 2024 at 2:14 AM Mia <mikasa.ack1618@gmail.com> wrote:
Thank you both; it was really helpful.
On Wed, Oct 16, 2024 at 11:05 PM Sean Mooney <smooney@redhat.com> wrote:
On 16/10/2024 19:49, Dmitriy Rabotyagov wrote:
Hey,
Somebody may correct me, but to the best of my knowledge this functionality is currently not implemented in Nova.
it was proposed, discused for a few release and ultimatly rejected.
https://blueprints.launchpad.net/nova/+spec/preemptible-instances https://review.opendev.org/c/openstack/nova-specs/+/438640 https://review.opendev.org/c/openstack/nova/+/547450
the most recen tspec that was aproved for this was
https://specs.openstack.org/openstack/nova-specs/specs/train/approved/introd...
and out of tree protype was developed
https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype
but it was never compelted.
after that did not pan out it moved to balzar
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
and again it never completed to my knowladge
I can assume there could be some custom scheduler filters/weighers in the wild (not 100% sure this can be done solely through scheduling though), but I personally haven't seen one so far. Though I was asked a couple of times during last year about exactly the same thing. So you're totally not alone looking for a solution for spot instances.
it is a potential usescase that could be supproted but the logic for doign the reaping/premetion is non trvial to do correctly.
one approch we have rejected in the past is havign nova's sechuler call out to another service if we cant find a host
or activly reaping premtiable instnace in that case.
On Wed, Oct 16, 2024, 14:09 Mia <mikasa.ack1618@gmail.com> wrote:
Hello everyone,
I am working on a project in the OpenStack environment. More precisely, I am looking into the implementation of spot instances as a cost optimization strategy and better resource utilization.
If any community members could share their experiences and insights related to the following areas, it would be greatly appreciated. Implementation Strategies: How did you go about implementing spot instances in your cluster? Challenges Faced: Mention some of the challenges or limitations. Best Practices: Do you have any recommendations or best practices that you would like to suggest for seamless deployment?
Thank you very much for your time and support.
We at Nectar still use Aardvark for preemptible instances. It is a fork of what CERN developed with some modifications. How we use is is we have flavours with a property that indicates the flavour is preemptible. Then aardvark will delete these Instances when capacity is over a threshold or 24 hours, which ever comes first. We then charge significantly less for these flavours. So the pricing isn’t dynamic like spot instances but it still allows some of our bigger users to chew up some free capacity. Source: https://github.com/NeCTAR-RC/aardvark We deploy ours with this helm chart https://github.com/NeCTAR-RC/aardvark-helm Cheers, Sam
On 18 Oct 2024, at 2:26 am, Arne Wiebalck <Arne.Wiebalck@cern.ch> wrote:
FWIW, we actually had a full implementation of pre-emptible instances: it changed name from Reaper to Aardvark and it was used in production here at CERN until a recent upgrade.
The design and patch sets were discussed with upstream at the time, but the required change of the state machine (to add the "pending" state) was the main point which prevented integration into the upstream code base eventually.
We do not use this functionality in our production deployment at the moment, mostly since we weighed carrying forward a local patch set versus the benefit of the added functionality, but the code is of course still there and we're happy to share it again if there is revived interest in the community.
Cheers, Arne
-- Arne Wiebalck CERN IT
________________________________________ From: Julia Kreger <juliaashleykreger@gmail.com> Sent: Thursday, 17 October 2024 16:23 To: Mia Cc: openstack-discuss@lists.openstack.org Subject: Re: Spot Instances
Just an observation following up from discussions I was part of as well as presentations I listened to at OpenInfra Days NA which occurred this week. There were a surprising number of mentions of spot instance-like functionality by some OpenStack vendors and scientific/academic infrastructure operators.
It might be good for the wider community to revisit such discussion, given next week is the PTG, as there is clearly renewed interest and different practices being adopted which creates an inconsistent user experience across different OpenStack clouds.
-Julia
On Thu, Oct 17, 2024 at 2:14 AM Mia <mikasa.ack1618@gmail.com> wrote:
Thank you both; it was really helpful.
On Wed, Oct 16, 2024 at 11:05 PM Sean Mooney <smooney@redhat.com> wrote:
On 16/10/2024 19:49, Dmitriy Rabotyagov wrote:
Hey,
Somebody may correct me, but to the best of my knowledge this functionality is currently not implemented in Nova.
it was proposed, discused for a few release and ultimatly rejected.
https://blueprints.launchpad.net/nova/+spec/preemptible-instances https://review.opendev.org/c/openstack/nova-specs/+/438640 https://review.opendev.org/c/openstack/nova/+/547450
the most recen tspec that was aproved for this was
https://specs.openstack.org/openstack/nova-specs/specs/train/approved/introd...
and out of tree protype was developed
https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype
but it was never compelted.
after that did not pan out it moved to balzar
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
and again it never completed to my knowladge
I can assume there could be some custom scheduler filters/weighers in the wild (not 100% sure this can be done solely through scheduling though), but I personally haven't seen one so far. Though I was asked a couple of times during last year about exactly the same thing. So you're totally not alone looking for a solution for spot instances.
it is a potential usescase that could be supproted but the logic for doign the reaping/premetion is non trvial to do correctly.
one approch we have rejected in the past is havign nova's sechuler call out to another service if we cant find a host
or activly reaping premtiable instnace in that case.
On Wed, Oct 16, 2024, 14:09 Mia <mikasa.ack1618@gmail.com> wrote:
Hello everyone,
I am working on a project in the OpenStack environment. More precisely, I am looking into the implementation of spot instances as a cost optimization strategy and better resource utilization.
If any community members could share their experiences and insights related to the following areas, it would be greatly appreciated. Implementation Strategies: How did you go about implementing spot instances in your cluster? Challenges Faced: Mention some of the challenges or limitations. Best Practices: Do you have any recommendations or best practices that you would like to suggest for seamless deployment?
Thank you very much for your time and support.
Hi Sam & Arne, This is really helpful. We are actually looking for similar solutions to experiment with. We'll try Aardvark in our lab and let you know if we need any assistance. We’d also be happy to contribute if the community considers bringing Aardvark back into the upstream. Regards Vivek On Fri, Oct 18, 2024 at 5:36 AM Sam Morrison <sorrison@gmail.com> wrote:
We at Nectar still use Aardvark for preemptible instances. It is a fork of what CERN developed with some modifications.
How we use is is we have flavours with a property that indicates the flavour is preemptible. Then aardvark will delete these Instances when capacity is over a threshold or 24 hours, which ever comes first.
We then charge significantly less for these flavours. So the pricing isn’t dynamic like spot instances but it still allows some of our bigger users to chew up some free capacity.
Source: https://github.com/NeCTAR-RC/aardvark
We deploy ours with this helm chart https://github.com/NeCTAR-RC/aardvark-helm
Cheers, Sam
On 18 Oct 2024, at 2:26 am, Arne Wiebalck <Arne.Wiebalck@cern.ch> wrote:
FWIW, we actually had a full implementation of pre-emptible instances: it changed name from Reaper to Aardvark and it was used in production here at CERN until a recent upgrade.
The design and patch sets were discussed with upstream at the time, but the required change of the state machine (to add the "pending" state) was the main point which prevented integration into the upstream code base eventually.
We do not use this functionality in our production deployment at the moment, mostly since we weighed carrying forward a local patch set versus the benefit of the added functionality, but the code is of course still there and we're happy to share it again if there is revived interest in the community.
Cheers, Arne
-- Arne Wiebalck CERN IT
________________________________________ From: Julia Kreger <juliaashleykreger@gmail.com> Sent: Thursday, 17 October 2024 16:23 To: Mia Cc: openstack-discuss@lists.openstack.org Subject: Re: Spot Instances
Just an observation following up from discussions I was part of as well as presentations I listened to at OpenInfra Days NA which occurred this week. There were a surprising number of mentions of spot instance-like functionality by some OpenStack vendors and scientific/academic infrastructure operators.
It might be good for the wider community to revisit such discussion, given next week is the PTG, as there is clearly renewed interest and different practices being adopted which creates an inconsistent user experience across different OpenStack clouds.
-Julia
On Thu, Oct 17, 2024 at 2:14 AM Mia <mikasa.ack1618@gmail.com> wrote:
Thank you both; it was really helpful.
On Wed, Oct 16, 2024 at 11:05 PM Sean Mooney <smooney@redhat.com> wrote:
On 16/10/2024 19:49, Dmitriy Rabotyagov wrote:
Hey,
Somebody may correct me, but to the best of my knowledge this functionality is currently not implemented in Nova.
it was proposed, discused for a few release and ultimatly rejected.
https://blueprints.launchpad.net/nova/+spec/preemptible-instances https://review.opendev.org/c/openstack/nova-specs/+/438640 https://review.opendev.org/c/openstack/nova/+/547450
the most recen tspec that was aproved for this was
https://specs.openstack.org/openstack/nova-specs/specs/train/approved/introd...
and out of tree protype was developed
https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype
but it was never compelted.
after that did not pan out it moved to balzar
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
and again it never completed to my knowladge
I can assume there could be some custom scheduler filters/weighers in the wild (not 100% sure this can be done solely through scheduling though), but I personally haven't seen one so far. Though I was asked a couple of times during last year about exactly the same thing. So you're totally not alone looking for a solution for spot instances.
it is a potential usescase that could be supproted but the logic for doign the reaping/premetion is non trvial to do correctly.
one approch we have rejected in the past is havign nova's sechuler call out to another service if we cant find a host
or activly reaping premtiable instnace in that case.
On Wed, Oct 16, 2024, 14:09 Mia <mikasa.ack1618@gmail.com> wrote:
Hello everyone,
I am working on a project in the OpenStack environment. More precisely, I am looking into the implementation of spot instances as a cost optimization strategy and better resource utilization.
If any community members could share their experiences and insights related to the following areas, it would be greatly appreciated. Implementation Strategies: How did you go about implementing spot instances in your cluster? Challenges Faced: Mention some of the challenges or limitations. Best Practices: Do you have any recommendations or best practices that you would like to suggest for seamless deployment?
Thank you very much for your time and support.
Hi everyone, Thank you for your helpful responses. I appreciate the information and resources you've shared about the implementation. We'll try them out and be very happy to contribute. On Fri, Oct 18, 2024 at 4:47 AM Vivekanandan Muthukrishnan < vivek@vnetpops.com> wrote:
Hi Sam & Arne,
This is really helpful. We are actually looking for similar solutions to experiment with. We'll try Aardvark in our lab and let you know if we need any assistance.
We’d also be happy to contribute if the community considers bringing Aardvark back into the upstream. Regards Vivek
On Fri, Oct 18, 2024 at 5:36 AM Sam Morrison <sorrison@gmail.com> wrote:
We at Nectar still use Aardvark for preemptible instances. It is a fork of what CERN developed with some modifications.
How we use is is we have flavours with a property that indicates the flavour is preemptible. Then aardvark will delete these Instances when capacity is over a threshold or 24 hours, which ever comes first.
We then charge significantly less for these flavours. So the pricing isn’t dynamic like spot instances but it still allows some of our bigger users to chew up some free capacity.
Source: https://github.com/NeCTAR-RC/aardvark
We deploy ours with this helm chart https://github.com/NeCTAR-RC/aardvark-helm
Cheers, Sam
On 18 Oct 2024, at 2:26 am, Arne Wiebalck <Arne.Wiebalck@cern.ch> wrote:
FWIW, we actually had a full implementation of pre-emptible instances: it changed name from Reaper to Aardvark and it was used in production here at CERN until a recent upgrade.
The design and patch sets were discussed with upstream at the time, but the required change of the state machine (to add the "pending" state) was the main point which prevented integration into the upstream code base eventually.
We do not use this functionality in our production deployment at the moment, mostly since we weighed carrying forward a local patch set versus the benefit of the added functionality, but the code is of course still there and we're happy to share it again if there is revived interest in the community.
Cheers, Arne
-- Arne Wiebalck CERN IT
________________________________________ From: Julia Kreger <juliaashleykreger@gmail.com> Sent: Thursday, 17 October 2024 16:23 To: Mia Cc: openstack-discuss@lists.openstack.org Subject: Re: Spot Instances
Just an observation following up from discussions I was part of as well as presentations I listened to at OpenInfra Days NA which occurred this week. There were a surprising number of mentions of spot instance-like functionality by some OpenStack vendors and scientific/academic infrastructure operators.
It might be good for the wider community to revisit such discussion, given next week is the PTG, as there is clearly renewed interest and different practices being adopted which creates an inconsistent user experience across different OpenStack clouds.
-Julia
On Thu, Oct 17, 2024 at 2:14 AM Mia <mikasa.ack1618@gmail.com> wrote:
Thank you both; it was really helpful.
On Wed, Oct 16, 2024 at 11:05 PM Sean Mooney <smooney@redhat.com> wrote:
On 16/10/2024 19:49, Dmitriy Rabotyagov wrote:
Hey,
Somebody may correct me, but to the best of my knowledge this functionality is currently not implemented in Nova.
it was proposed, discused for a few release and ultimatly rejected.
https://blueprints.launchpad.net/nova/+spec/preemptible-instances https://review.opendev.org/c/openstack/nova-specs/+/438640 https://review.opendev.org/c/openstack/nova/+/547450
the most recen tspec that was aproved for this was
https://specs.openstack.org/openstack/nova-specs/specs/train/approved/introd...
and out of tree protype was developed
https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype
but it was never compelted.
after that did not pan out it moved to balzar
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
and again it never completed to my knowladge
I can assume there could be some custom scheduler filters/weighers in the wild (not 100% sure this can be done solely through scheduling though), but I personally haven't seen one so far. Though I was asked a couple of times during last year about exactly the same thing. So you're totally not alone looking for a solution for spot instances.
it is a potential usescase that could be supproted but the logic for doign the reaping/premetion is non trvial to do correctly.
one approch we have rejected in the past is havign nova's sechuler call out to another service if we cant find a host
or activly reaping premtiable instnace in that case.
On Wed, Oct 16, 2024, 14:09 Mia <mikasa.ack1618@gmail.com> wrote:
Hello everyone,
I am working on a project in the OpenStack environment. More precisely, I am looking into the implementation of spot instances as a cost optimization strategy and better resource utilization.
If any community members could share their experiences and insights related to the following areas, it would be greatly appreciated. Implementation Strategies: How did you go about implementing spot instances in your cluster? Challenges Faced: Mention some of the challenges or limitations. Best Practices: Do you have any recommendations or best practices that you would like to suggest for seamless deployment?
Thank you very much for your time and support.
participants (8)
-
Arne Wiebalck
-
Dmitriy Rabotyagov
-
Julia Kreger
-
Karl Kloppenborg
-
Mia
-
Sam Morrison
-
Sean Mooney
-
Vivekanandan Muthukrishnan