[magnum] GPU support with magnum
Folks, We are running magnum with openstack to spin up k8s clusters and everything works as expected. Now I want GPU support so we can do some AI/ML stuff with k8s. Does magnum support GPU and how does it work? In the past I used GPU in passthrough mode to expose hardware to VMs but not sure how it works for k8s and what are the challenges?
Hi Sathish, I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs. You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now. I planned to write a blog post from my personal notes, but it is not finished yet. Cheers, Oliver Am 07.01.2025 um 16:07 schrieb Satish Patel:
Folks,
We are running magnum with openstack to spin up k8s clusters and everything works as expected. Now I want GPU support so we can do some AI/ML stuff with k8s. Does magnum support GPU and how does it work?
In the past I used GPU in passthrough mode to expose hardware to VMs but not sure how it works for k8s and what are the challenges?
Hi Oliver, Thank you so much for your reply! Please make a blog :) I am very interested to see. On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Sathish,
I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs.
You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now.
I planned to write a blog post from my personal notes, but it is not finished yet.
Cheers,
Oliver
Am 07.01.2025 um 16:07 schrieb Satish Patel:
Folks,
We are running magnum with openstack to spin up k8s clusters and everything works as expected. Now I want GPU support so we can do some AI/ML stuff with k8s. Does magnum support GPU and how does it work?
In the past I used GPU in passthrough mode to expose hardware to VMs but not sure how it works for k8s and what are the challenges?
Hi Satish, All done. Since you already have GPU Passthrough working, this should be easy. https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum... I hope I got everything right. It has been a while and I have now free GPU at the moment to test all the steps again. Please let me know in case you need help. Cheers, Oliver Am 07.01.2025 um 20:44 schrieb Satish Patel:
Hi Oliver,
Thank you so much for your reply! Please make a blog :) I am very interested to see.
On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Sathish,
I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs.
You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now.
I planned to write a blog post from my personal notes, but it is not finished yet.
Cheers,
Oliver
Am 07.01.2025 um 16:07 schrieb Satish Patel: > Folks, > > We are running magnum with openstack to spin up k8s clusters and > everything works as expected. Now I want GPU support so we can do some > AI/ML stuff with k8s. Does magnum support GPU and how does it work? > > In the past I used GPU in passthrough mode to expose hardware to VMs > but not sure how it works for k8s and what are the challenges?
Hi Oliver, It says "The post is for subscribers only" and I got an error when trying to sign up. On Tue, Jan 7, 2025 at 4:10 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
All done.
Since you already have GPU Passthrough working, this should be easy.
https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum...
I hope I got everything right. It has been a while and I have now free GPU at the moment to test all the steps again. Please let me know in case you need help.
Cheers,
Oliver Am 07.01.2025 um 20:44 schrieb Satish Patel:
Hi Oliver,
Thank you so much for your reply! Please make a blog :) I am very interested to see.
On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Sathish,
I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs.
You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now.
I planned to write a blog post from my personal notes, but it is not finished yet.
Cheers,
Oliver
Am 07.01.2025 um 16:07 schrieb Satish Patel:
Folks,
We are running magnum with openstack to spin up k8s clusters and everything works as expected. Now I want GPU support so we can do some AI/ML stuff with k8s. Does magnum support GPU and how does it work?
In the past I used GPU in passthrough mode to expose hardware to VMs but not sure how it works for k8s and what are the challenges?
Try this one: https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... easily reachable from the previous link C On Wed, Jan 8, 2025 at 3:44 AM Satish Patel <satish.txt@gmail.com> wrote:
Hi Oliver,
It says "The post is for subscribers only" and I got an error when trying to sign up.
On Tue, Jan 7, 2025 at 4:10 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
All done.
Since you already have GPU Passthrough working, this should be easy.
https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum...
I hope I got everything right. It has been a while and I have now free GPU at the moment to test all the steps again. Please let me know in case you need help.
Cheers,
Oliver Am 07.01.2025 um 20:44 schrieb Satish Patel:
Hi Oliver,
Thank you so much for your reply! Please make a blog :) I am very interested to see.
On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Sathish,
I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs.
You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now.
I planned to write a blog post from my personal notes, but it is not finished yet.
Cheers,
Oliver
Am 07.01.2025 um 16:07 schrieb Satish Patel:
Folks,
We are running magnum with openstack to spin up k8s clusters and everything works as expected. Now I want GPU support so we can do some AI/ML stuff with k8s. Does magnum support GPU and how does it work?
In the past I used GPU in passthrough mode to expose hardware to VMs but not sure how it works for k8s and what are the challenges?
Hi Oliver, Here is the error when I sign up - https://ibb.co/ZBgZktd @Doina - This link is to enable GPU on openstack but not magnum - https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... On Wed, Jan 8, 2025 at 4:06 AM Doina Cristina Duma <caifti@gmail.com> wrote:
Try this one:
https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... easily reachable from the previous link
C
On Wed, Jan 8, 2025 at 3:44 AM Satish Patel <satish.txt@gmail.com> wrote:
Hi Oliver,
It says "The post is for subscribers only" and I got an error when trying to sign up.
On Tue, Jan 7, 2025 at 4:10 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
All done.
Since you already have GPU Passthrough working, this should be easy.
https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum...
I hope I got everything right. It has been a while and I have now free GPU at the moment to test all the steps again. Please let me know in case you need help.
Cheers,
Oliver Am 07.01.2025 um 20:44 schrieb Satish Patel:
Hi Oliver,
Thank you so much for your reply! Please make a blog :) I am very interested to see.
On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Sathish,
I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs.
You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now.
I planned to write a blog post from my personal notes, but it is not finished yet.
Cheers,
Oliver
Am 07.01.2025 um 16:07 schrieb Satish Patel:
Folks,
We are running magnum with openstack to spin up k8s clusters and everything works as expected. Now I want GPU support so we can do some AI/ML stuff with k8s. Does magnum support GPU and how does it work?
In the past I used GPU in passthrough mode to expose hardware to VMs but not sure how it works for k8s and what are the challenges?
Hi Satish, confirmed. I was able to reproduce your error. Seems it has to do with gmail googlemail. I will look into this. For now the blogpost is publicly accessible. :) Cheers, Oliver Am 08.01.2025 um 15:10 schrieb Satish Patel:
Hi Oliver,
Here is the error when I sign up - https://ibb.co/ZBgZktd
@Doina - This link is to enable GPU on openstack but not magnum - https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-...
On Wed, Jan 8, 2025 at 4:06 AM Doina Cristina Duma <caifti@gmail.com> wrote:
Try this one: https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-...
easily reachable from the previous link
C
On Wed, Jan 8, 2025 at 3:44 AM Satish Patel <satish.txt@gmail.com> wrote:
Hi Oliver,
It says "The post is for subscribers only" and I got an error when trying to sign up.
On Tue, Jan 7, 2025 at 4:10 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
All done.
Since you already have GPU Passthrough working, this should be easy.
https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum...
I hope I got everything right. It has been a while and I have now free GPU at the moment to test all the steps again. Please let me know in case you need help.
Cheers,
Oliver
Am 07.01.2025 um 20:44 schrieb Satish Patel:
Hi Oliver,
Thank you so much for your reply! Please make a blog :) I am very interested to see.
On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Sathish,
I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs.
You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now.
I planned to write a blog post from my personal notes, but it is not finished yet.
Cheers,
Oliver
Am 07.01.2025 um 16:07 schrieb Satish Patel: > Folks, > > We are running magnum with openstack to spin up k8s clusters and > everything works as expected. Now I want GPU support so we can do some > AI/ML stuff with k8s. Does magnum support GPU and how does it work? > > In the past I used GPU in passthrough mode to expose hardware to VMs > but not sure how it works for k8s and what are the challenges?
Awesome, thank you so much! On Wed, Jan 8, 2025 at 10:15 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
confirmed. I was able to reproduce your error. Seems it has to do with gmail googlemail. I will look into this. For now the blogpost is publicly accessible. :)
Cheers,
Oliver Am 08.01.2025 um 15:10 schrieb Satish Patel:
Hi Oliver,
Here is the error when I sign up - https://ibb.co/ZBgZktd
@Doina - This link is to enable GPU on openstack but not magnum - https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-...
On Wed, Jan 8, 2025 at 4:06 AM Doina Cristina Duma <caifti@gmail.com> wrote:
Try this one:
https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... easily reachable from the previous link
C
On Wed, Jan 8, 2025 at 3:44 AM Satish Patel <satish.txt@gmail.com> wrote:
Hi Oliver,
It says "The post is for subscribers only" and I got an error when trying to sign up.
On Tue, Jan 7, 2025 at 4:10 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
All done.
Since you already have GPU Passthrough working, this should be easy.
https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum...
I hope I got everything right. It has been a while and I have now free GPU at the moment to test all the steps again. Please let me know in case you need help.
Cheers,
Oliver Am 07.01.2025 um 20:44 schrieb Satish Patel:
Hi Oliver,
Thank you so much for your reply! Please make a blog :) I am very interested to see.
On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Sathish,
I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs.
You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now.
I planned to write a blog post from my personal notes, but it is not finished yet.
Cheers,
Oliver
Am 07.01.2025 um 16:07 schrieb Satish Patel:
Folks,
We are running magnum with openstack to spin up k8s clusters and everything works as expected. Now I want GPU support so we can do some AI/ML stuff with k8s. Does magnum support GPU and how does it work?
In the past I used GPU in passthrough mode to expose hardware to VMs but not sure how it works for k8s and what are the challenges?
Hi Oliver The link below is not accessible. https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... Any help on this? I need to configure the GPU in my openstack cluster. Regards Keshav Bareja On Wed, Jan 8, 2025 at 3:16 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
confirmed. I was able to reproduce your error. Seems it has to do with gmail googlemail. I will look into this. For now the blogpost is publicly accessible. :)
Cheers,
Oliver Am 08.01.2025 um 15:10 schrieb Satish Patel:
Hi Oliver,
Here is the error when I sign up - https://ibb.co/ZBgZktd
@Doina - This link is to enable GPU on openstack but not magnum - https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-...
On Wed, Jan 8, 2025 at 4:06 AM Doina Cristina Duma <caifti@gmail.com> wrote:
Try this one:
https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... easily reachable from the previous link
C
On Wed, Jan 8, 2025 at 3:44 AM Satish Patel <satish.txt@gmail.com> wrote:
Hi Oliver,
It says "The post is for subscribers only" and I got an error when trying to sign up.
On Tue, Jan 7, 2025 at 4:10 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
All done.
Since you already have GPU Passthrough working, this should be easy.
https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum...
I hope I got everything right. It has been a while and I have now free GPU at the moment to test all the steps again. Please let me know in case you need help.
Cheers,
Oliver Am 07.01.2025 um 20:44 schrieb Satish Patel:
Hi Oliver,
Thank you so much for your reply! Please make a blog :) I am very interested to see.
On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Sathish,
I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs.
You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now.
I planned to write a blog post from my personal notes, but it is not finished yet.
Cheers,
Oliver
Am 07.01.2025 um 16:07 schrieb Satish Patel:
Folks,
We are running magnum with openstack to spin up k8s clusters and everything works as expected. Now I want GPU support so we can do some AI/ML stuff with k8s. Does magnum support GPU and how does it work?
In the past I used GPU in passthrough mode to expose hardware to VMs but not sure how it works for k8s and what are the challenges?
Hi Oliver, Sorry to hear that, I didn't know you are running your blog on k8s :o You had no backup at all? That is why I keep my blog on github with a plain text file to keep it simple and less worry to take backup. Hope you get backup from somewhere. I badly need some posts. On Fri, Jan 17, 2025 at 7:28 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Keshav,
Unfortunately, I killed my whole blog by mistake last weekend.
* https://www.roksblog.de/i-screwed-up-big-time/ <https://www.roksblog.de/i-screwed-up-big-time/>
I was not able to recover all posts.
I’m planning on re-writing this blog post, maybe later today.
I know there was at least one user from the mailing list that read it. Maybe he took some notes?
Cheers, Oliver
Von meinem iPhone gesendet
Am 17.01.2025 um 12:58 schrieb keshav bareja <keshav.bareja@gmail.com>:
Hi Oliver
The link below is not accessible.
https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-...
Any help on this? I need to configure the GPU in my openstack cluster.
Regards Keshav Bareja
On Wed, Jan 8, 2025 at 3:16 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
confirmed. I was able to reproduce your error. Seems it has to do with gmail googlemail. I will look into this. For now the blogpost is publicly accessible. :)
Cheers,
Oliver Am 08.01.2025 um 15:10 schrieb Satish Patel:
Hi Oliver,
Here is the error when I sign up - https://ibb.co/ZBgZktd
@Doina - This link is to enable GPU on openstack but not magnum - https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-...
On Wed, Jan 8, 2025 at 4:06 AM Doina Cristina Duma <caifti@gmail.com> wrote:
Try this one:
https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... easily reachable from the previous link
C
On Wed, Jan 8, 2025 at 3:44 AM Satish Patel <satish.txt@gmail.com> wrote:
Hi Oliver,
It says "The post is for subscribers only" and I got an error when trying to sign up.
On Tue, Jan 7, 2025 at 4:10 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
All done.
Since you already have GPU Passthrough working, this should be easy.
https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum...
I hope I got everything right. It has been a while and I have now free GPU at the moment to test all the steps again. Please let me know in case you need help.
Cheers,
Oliver Am 07.01.2025 um 20:44 schrieb Satish Patel:
Hi Oliver,
Thank you so much for your reply! Please make a blog :) I am very interested to see.
On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann < oliver.weinmann@me.com> wrote:
Hi Sathish,
I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs.
You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now.
I planned to write a blog post from my personal notes, but it is not finished yet.
Cheers,
Oliver
Am 07.01.2025 um 16:07 schrieb Satish Patel: > Folks, > > We are running magnum with openstack to spin up k8s clusters and > everything works as expected. Now I want GPU support so we can do some > AI/ML stuff with k8s. Does magnum support GPU and how does it work? > > In the past I used GPU in passthrough mode to expose hardware to VMs > but not sure how it works for k8s and what are the challenges?
Hi all, post is now back online https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... This completes my 3 post series about GPUs in OpenStack Magnum. Have a nice weekend Am 17.01.2025 um 15:20 schrieb Satish Patel:
Hi Oliver,
Sorry to hear that, I didn't know you are running your blog on k8s :o You had no backup at all? That is why I keep my blog on github with a plain text file to keep it simple and less worry to take backup.
Hope you get backup from somewhere. I badly need some posts.
On Fri, Jan 17, 2025 at 7:28 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Keshav,
Unfortunately, I killed my whole blog by mistake last weekend.
* https://www.roksblog.de/i-screwed-up-big-time/ <https://www.roksblog.de/i-screwed-up-big-time/>
I was not able to recover all posts.
I’m planning on re-writing this blog post, maybe later today.
I know there was at least one user from the mailing list that read it. Maybe he took some notes?
Cheers, Oliver
Von meinem iPhone gesendet
Am 17.01.2025 um 12:58 schrieb keshav bareja <keshav.bareja@gmail.com>:
Hi Oliver
The link below is not accessible.
https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-...
Any help on this? I need to configure the GPU in my openstack cluster.
Regards Keshav Bareja
On Wed, Jan 8, 2025 at 3:16 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
confirmed. I was able to reproduce your error. Seems it has to do with gmail googlemail. I will look into this. For now the blogpost is publicly accessible. :)
Cheers,
Oliver
Am 08.01.2025 um 15:10 schrieb Satish Patel:
Hi Oliver,
Here is the error when I sign up - https://ibb.co/ZBgZktd
@Doina - This link is to enable GPU on openstack but not magnum - https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-...
On Wed, Jan 8, 2025 at 4:06 AM Doina Cristina Duma <caifti@gmail.com> wrote:
Try this one: https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-...
easily reachable from the previous link
C
On Wed, Jan 8, 2025 at 3:44 AM Satish Patel <satish.txt@gmail.com> wrote:
Hi Oliver,
It says "The post is for subscribers only" and I got an error when trying to sign up.
On Tue, Jan 7, 2025 at 4:10 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
All done.
Since you already have GPU Passthrough working, this should be easy.
https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum...
I hope I got everything right. It has been a while and I have now free GPU at the moment to test all the steps again. Please let me know in case you need help.
Cheers,
Oliver
Am 07.01.2025 um 20:44 schrieb Satish Patel:
Hi Oliver,
Thank you so much for your reply! Please make a blog :) I am very interested to see.
On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Sathish,
I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs.
You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now.
I planned to write a blog post from my personal notes, but it is not finished yet.
Cheers,
Oliver
Am 07.01.2025 um 16:07 schrieb Satish Patel: > Folks, > > We are running magnum with openstack to spin up k8s clusters and > everything works as expected. Now I want GPU support so we can do some > AI/ML stuff with k8s. Does magnum support GPU and how does it work? > > In the past I used GPU in passthrough mode to expose hardware to VMs > but not sure how it works for k8s and what are the challenges?
Hi Oliver: https://web.archive.org/web/20240915000000*/https://www.roksblog.de/ Perhaps this will be of help 🙂 Good luck and thansk for the cool content. Mohammed ________________________________ From: Oliver Weinmann <oliver.weinmann@me.com> Sent: January 17, 2025 4:31 PM To: Satish Patel <satish.txt@gmail.com> Cc: keshav bareja <keshav.bareja@gmail.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: Re: [magnum] GPU support with magnum Hi all, post is now back online https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... This completes my 3 post series about GPUs in OpenStack Magnum. Have a nice weekend Am 17.01.2025 um 15:20 schrieb Satish Patel: Hi Oliver, Sorry to hear that, I didn't know you are running your blog on k8s :o You had no backup at all? That is why I keep my blog on github with a plain text file to keep it simple and less worry to take backup. Hope you get backup from somewhere. I badly need some posts. On Fri, Jan 17, 2025 at 7:28 AM Oliver Weinmann <oliver.weinmann@me.com<mailto:oliver.weinmann@me.com>> wrote: Hi Keshav, Unfortunately, I killed my whole blog by mistake last weekend. * https://www.roksblog.de/i-screwed-up-big-time/<https://www.roksblog.de/i-screwed-up-big-time/> I was not able to recover all posts. I’m planning on re-writing this blog post, maybe later today. I know there was at least one user from the mailing list that read it. Maybe he took some notes? Cheers, Oliver Von meinem iPhone gesendet Am 17.01.2025 um 12:58 schrieb keshav bareja <keshav.bareja@gmail.com<mailto:keshav.bareja@gmail.com>>: Hi Oliver The link below is not accessible. https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... Any help on this? I need to configure the GPU in my openstack cluster. Regards Keshav Bareja On Wed, Jan 8, 2025 at 3:16 PM Oliver Weinmann <oliver.weinmann@me.com<mailto:oliver.weinmann@me.com>> wrote: Hi Satish, confirmed. I was able to reproduce your error. Seems it has to do with gmail googlemail. I will look into this. For now the blogpost is publicly accessible. :) Cheers, Oliver Am 08.01.2025 um 15:10 schrieb Satish Patel: Hi Oliver, Here is the error when I sign up - https://ibb.co/ZBgZktd @Doina - This link is to enable GPU on openstack but not magnum - https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... On Wed, Jan 8, 2025 at 4:06 AM Doina Cristina Duma <caifti@gmail.com<mailto:caifti@gmail.com>> wrote: Try this one: https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... easily reachable from the previous link C On Wed, Jan 8, 2025 at 3:44 AM Satish Patel <satish.txt@gmail.com<mailto:satish.txt@gmail.com>> wrote: Hi Oliver, It says "The post is for subscribers only" and I got an error when trying to sign up. On Tue, Jan 7, 2025 at 4:10 PM Oliver Weinmann <oliver.weinmann@me.com<mailto:oliver.weinmann@me.com>> wrote: Hi Satish, All done. Since you already have GPU Passthrough working, this should be easy. https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum... I hope I got everything right. It has been a while and I have now free GPU at the moment to test all the steps again. Please let me know in case you need help. Cheers, Oliver Am 07.01.2025 um 20:44 schrieb Satish Patel: Hi Oliver, Thank you so much for your reply! Please make a blog :) I am very interested to see. On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann <oliver.weinmann@me.com<mailto:oliver.weinmann@me.com>> wrote: Hi Sathish, I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs. You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now. I planned to write a blog post from my personal notes, but it is not finished yet. Cheers, Oliver Am 07.01.2025 um 16:07 schrieb Satish Patel:
Folks,
We are running magnum with openstack to spin up k8s clusters and everything works as expected. Now I want GPU support so we can do some AI/ML stuff with k8s. Does magnum support GPU and how does it work?
In the past I used GPU in passthrough mode to expose hardware to VMs but not sure how it works for k8s and what are the challenges?
Hi Mohammed, thanks for the tip. That is actually how I recovered almost all of my posts. Just the latest ones and the ones not publicly available are lost. K8s Backup and Restore is now working as expected. Lesson learned. Thanks, glad to contribute. :) Cheers, Oliver Am 17.01.2025 um 22:33 schrieb Mohammed Naser:
Hi Oliver:
https://web.archive.org/web/20240915000000*/https://www.roksblog.de/
Perhaps this will be of help 🙂
Good luck and thansk for the cool content.
Mohammed
------------------------------------------------------------------------ *From:* Oliver Weinmann <oliver.weinmann@me.com> *Sent:* January 17, 2025 4:31 PM *To:* Satish Patel <satish.txt@gmail.com> *Cc:* keshav bareja <keshav.bareja@gmail.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org> *Subject:* Re: [magnum] GPU support with magnum
Hi all,
post is now back online
https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... <https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-for-high-performance-workloads/>
This completes my 3 post series about GPUs in OpenStack Magnum.
Have a nice weekend
Am 17.01.2025 um 15:20 schrieb Satish Patel:
Hi Oliver,
Sorry to hear that, I didn't know you are running your blog on k8s :o You had no backup at all? That is why I keep my blog on github with a plain text file to keep it simple and less worry to take backup.
Hope you get backup from somewhere. I badly need some posts.
On Fri, Jan 17, 2025 at 7:28 AM Oliver Weinmann <oliver.weinmann@me.com <mailto:oliver.weinmann@me.com>> wrote:
Hi Keshav,
Unfortunately, I killed my whole blog by mistake last weekend.
* https://www.roksblog.de/i-screwed-up-big-time/ <https://www.roksblog.de/i-screwed-up-big-time/>
I was not able to recover all posts.
I’m planning on re-writing this blog post, maybe later today.
I know there was at least one user from the mailing list that read it. Maybe he took some notes?
Cheers, Oliver
Von meinem iPhone gesendet
Am 17.01.2025 um 12:58 schrieb keshav bareja <keshav.bareja@gmail.com <mailto:keshav.bareja@gmail.com>>:
Hi Oliver
The link below is not accessible.
https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... <https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-for-high-performance-workloads/>
Any help on this? I need to configure the GPU in my openstack cluster.
Regards Keshav Bareja
On Wed, Jan 8, 2025 at 3:16 PM Oliver Weinmann <oliver.weinmann@me.com <mailto:oliver.weinmann@me.com>> wrote:
Hi Satish,
confirmed. I was able to reproduce your error. Seems it has to do with gmail googlemail. I will look into this. For now the blogpost is publicly accessible. :)
Cheers,
Oliver
Am 08.01.2025 um 15:10 schrieb Satish Patel:
Hi Oliver,
Here is the error when I sign up - https://ibb.co/ZBgZktd <https://ibb.co/ZBgZktd>
@Doina - This link is to enable GPU on openstack but not magnum - https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... <https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-for-high-performance-workloads/>
On Wed, Jan 8, 2025 at 4:06 AM Doina Cristina Duma <caifti@gmail.com <mailto:caifti@gmail.com>> wrote:
Try this one: https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... <https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-for-high-performance-workloads/>
easily reachable from the previous link
C
On Wed, Jan 8, 2025 at 3:44 AM Satish Patel <satish.txt@gmail.com <mailto:satish.txt@gmail.com>> wrote:
Hi Oliver,
It says "The post is for subscribers only" and I got an error when trying to sign up.
On Tue, Jan 7, 2025 at 4:10 PM Oliver Weinmann <oliver.weinmann@me.com <mailto:oliver.weinmann@me.com>> wrote:
Hi Satish,
All done.
Since you already have GPU Passthrough working, this should be easy.
https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum... <https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum-with-kolla-ansible/>
I hope I got everything right. It has been a while and I have now free GPU at the moment to test all the steps again. Please let me know in case you need help.
Cheers,
Oliver
Am 07.01.2025 um 20:44 schrieb Satish Patel:
Hi Oliver,
Thank you so much for your reply! Please make a blog :) I am very interested to see.
On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann <oliver.weinmann@me.com <mailto:oliver.weinmann@me.com>> wrote:
Hi Sathish,
I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs.
You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now.
I planned to write a blog post from my personal notes, but it is not finished yet.
Cheers,
Oliver
Am 07.01.2025 um 16:07 schrieb Satish Patel: > Folks, > > We are running magnum with openstack to spin up k8s clusters and > everything works as expected. Now I want GPU support so we can do some > AI/ML stuff with k8s. Does magnum support GPU and how does it work? > > In the past I used GPU in passthrough mode to expose hardware to VMs > but not sure how it works for k8s and what are the challenges?
Yey!!! Finally. Now make one more blog related how to backup k8s and restore backup.. hehe On Sat, Jan 18, 2025 at 7:01 AM keshav bareja <keshav.bareja@gmail.com> wrote:
Hi Oliver
Thanks alot for restoring the blog.
Br/ Keshav
Sent from my iPhone
On 18 Jan 2025, at 10:04 AM, Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Mohammed,
thanks for the tip. That is actually how I recovered almost all of my posts. Just the latest ones and the ones not publicly available are lost.
K8s Backup and Restore is now working as expected. Lesson learned.
Thanks, glad to contribute. :)
Cheers,
Oliver
Am 17.01.2025 um 22:33 schrieb Mohammed Naser:
Hi Oliver:
https://web.archive.org/web/20240915000000*/https://www.roksblog.de/
Perhaps this will be of help 🙂
Good luck and thansk for the cool content.
Mohammed
------------------------------ *From:* Oliver Weinmann <oliver.weinmann@me.com> <oliver.weinmann@me.com> *Sent:* January 17, 2025 4:31 PM *To:* Satish Patel <satish.txt@gmail.com> <satish.txt@gmail.com> *Cc:* keshav bareja <keshav.bareja@gmail.com> <keshav.bareja@gmail.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org> <openstack-discuss@lists.openstack.org> *Subject:* Re: [magnum] GPU support with magnum
Hi all,
post is now back online
https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-...
This completes my 3 post series about GPUs in OpenStack Magnum.
Have a nice weekend Am 17.01.2025 um 15:20 schrieb Satish Patel:
Hi Oliver,
Sorry to hear that, I didn't know you are running your blog on k8s :o You had no backup at all? That is why I keep my blog on github with a plain text file to keep it simple and less worry to take backup.
Hope you get backup from somewhere. I badly need some posts.
On Fri, Jan 17, 2025 at 7:28 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Keshav,
Unfortunately, I killed my whole blog by mistake last weekend.
* https://www.roksblog.de/i-screwed-up-big-time/ <https://www.roksblog.de/i-screwed-up-big-time/>
I was not able to recover all posts.
I’m planning on re-writing this blog post, maybe later today.
I know there was at least one user from the mailing list that read it. Maybe he took some notes?
Cheers, Oliver
Von meinem iPhone gesendet
Am 17.01.2025 um 12:58 schrieb keshav bareja <keshav.bareja@gmail.com>:
Hi Oliver
The link below is not accessible.
https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-...
Any help on this? I need to configure the GPU in my openstack cluster.
Regards Keshav Bareja
On Wed, Jan 8, 2025 at 3:16 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
confirmed. I was able to reproduce your error. Seems it has to do with gmail googlemail. I will look into this. For now the blogpost is publicly accessible. :)
Cheers,
Oliver Am 08.01.2025 um 15:10 schrieb Satish Patel:
Hi Oliver,
Here is the error when I sign up - https://ibb.co/ZBgZktd
@Doina - This link is to enable GPU on openstack but not magnum - https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-...
On Wed, Jan 8, 2025 at 4:06 AM Doina Cristina Duma <caifti@gmail.com> wrote:
Try this one:
https://www.roksblog.de/enabling-gpu-passthrough-in-openstack-kolla-ansible-... easily reachable from the previous link
C
On Wed, Jan 8, 2025 at 3:44 AM Satish Patel <satish.txt@gmail.com> wrote:
Hi Oliver,
It says "The post is for subscribers only" and I got an error when trying to sign up.
On Tue, Jan 7, 2025 at 4:10 PM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Satish,
All done.
Since you already have GPU Passthrough working, this should be easy.
https://www.roksblog.de/unlocking-gpu-powered-kubernetes-in-openstack-magnum...
I hope I got everything right. It has been a while and I have now free GPU at the moment to test all the steps again. Please let me know in case you need help.
Cheers,
Oliver Am 07.01.2025 um 20:44 schrieb Satish Patel:
Hi Oliver,
Thank you so much for your reply! Please make a blog :) I am very interested to see.
On Tue, Jan 7, 2025 at 10:41 AM Oliver Weinmann <oliver.weinmann@me.com> wrote:
Hi Sathish,
I have this running with Magnum since a couple of months now. What I can say is that it is working. Magnum itself is not in any way aware of the GPUs.
You basically just need to set up GPU passthrough on at least one compute node, have a flavor with the GPU and add an additional node group to your k8s clusters with this flavor. It works very well when using CAPI (Ubuntu) since you can simply deploy NVIDIA GPU Operator HELM chart. This automatically detects your NVIDIA GPUs, installs drivers etc. and simply works. I also managed to get it working with FCOS but this involves some more manual work, since the NVIDIA GPU Operator HELM chart is not 100% compatible with FCOS. You need to manually install the nvidia GPU driver in the running worker node. At least that was the state back in summer 2024. Maybe it is compatible now.
I planned to write a blog post from my personal notes, but it is not finished yet.
Cheers,
Oliver
Am 07.01.2025 um 16:07 schrieb Satish Patel:
Folks,
We are running magnum with openstack to spin up k8s clusters and everything works as expected. Now I want GPU support so we can do some AI/ML stuff with k8s. Does magnum support GPU and how does it work?
In the past I used GPU in passthrough mode to expose hardware to VMs but not sure how it works for k8s and what are the challenges?
participants (5)
-
Doina Cristina Duma
-
keshav bareja
-
Mohammed Naser
-
Oliver Weinmann
-
Satish Patel