Create OpenStack VMs in few seconds
Hi,
I was looking for a way to provision and connect to VMs within a few seconds and ended up in the OpenStack Summit demo on 'OpenStack Virtual Machine Quickly Available in One Second'.
I highly appreciate if someone can provide a tutorial or related documentation on how to achieve this?
Regards, Danishka
This is a demo of a third party extention that was never upstreamed.
nova does not support create a base vm and then doing a local live migration or restore for memory snapshots to create another vm.
this approch likely has several security implciations that would not be accpeatable in a multi tenant enviornment.
we have disucssed this type of vm creation in the past and determined that it is not a valid implematnion of spawn. a virt driver that precreate vms or copys an existing instance can be faster but that virt driver is not considered a compliant implementation.
so in short there is no way to achive this today in a compliant openstack powered cloud.
On 25/03/2021 12:21, open infra wrote:
Hi,
I was looking for a way to provision and connect to VMs within a few seconds and ended up in the OpenStack Summit demo on 'OpenStack Virtual Machine Quickly Available in One Second'.
I highly appreciate if someone can provide a tutorial or related documentation on how to achieve this?
Regards, Danishka
On 25/03/2021 14:47, Sean Mooney wrote:
This is a demo of a third party extention that was never upstreamed.
nova does not support create a base vm and then doing a local live migration or restore for memory snapshots to create another vm.
this approch likely has several security implciations that would not be accpeatable in a multi tenant enviornment.
we have disucssed this type of vm creation in the past and determined that it is not a valid implematnion of spawn. a virt driver that precreate vms or copys an existing instance can be faster but that virt driver is not considered a compliant implementation.
just to follow up on this see the note on vm launch
https://docs.openstack.org/nova/latest/user/support-matrix.html#operation_la...
*Launch instance*
*Status: mandatory. *
*Notes: *Importing pre-existing running virtual machines on a host is considered out of scope of the cloud paradigm. Therefore this operation is mandatory to support in drivers.
the type of fast boot descibed in that video is not a valid implemation of creating a server. instance.
so in short there is no way to achive this today in a compliant openstack powered cloud.
On 25/03/2021 12:21, open infra wrote:
Hi,
I was looking for a way to provision and connect to VMs within a few seconds and ended up in the OpenStack Summit demo on 'OpenStack Virtual Machine Quickly Available in One Second'.
I highly appreciate if someone can provide a tutorial or related documentation on how to achieve this?
Regards, Danishka
On 2021-03-25 14:47:09 +0000 (+0000), Sean Mooney wrote:
nova does not support create a base vm and then doing a local live migration or restore for memory snapshots to create another vm.
this approch likely has several security implciations that would not be accpeatable in a multi tenant enviornment.
we have disucssed this type of vm creation in the past and determined that it is not a valid implematnion of spawn. a virt driver that precreate vms or copys an existing instance can be faster but that virt driver is not considered a compliant implementation.
so in short there is no way to achive this today in a compliant openstack powered cloud.
[...]
The next best thing is basically what Nodepool[*] does: start new virtual machines ahead of time and keep them available in the tenant. This does of course mean you're occupying additional quota for whatever base "ready" capacity you've set for your various images/flavors, and that you need to be able to predict how many of what kinds of virtual machines you're going to need in advance.
[*] https://zuul-ci.org/docs/nodepool/
On 25/03/2021 15:04, Jeremy Stanley wrote:
On 2021-03-25 14:47:09 +0000 (+0000), Sean Mooney wrote:
nova does not support create a base vm and then doing a local live migration or restore for memory snapshots to create another vm.
this approch likely has several security implciations that would not be accpeatable in a multi tenant enviornment.
we have disucssed this type of vm creation in the past and determined that it is not a valid implematnion of spawn. a virt driver that precreate vms or copys an existing instance can be faster but that virt driver is not considered a compliant implementation.
so in short there is no way to achive this today in a compliant openstack powered cloud.
[...]
The next best thing is basically what Nodepool[*] does: start new virtual machines ahead of time and keep them available in the tenant. This does of course mean you're occupying additional quota for whatever base "ready" capacity you've set for your various images/flavors, and that you need to be able to predict how many of what kinds of virtual machines you're going to need in advance.
ya if you had premtible instances that would make that model a little more compeeling especially if you could move them form premtible to non premtable wehn you acutlly use them.
unfortunetly we also dont have the feature in nova today.
if the extra cost of precreating the instances was acceptable then yes using something like node pool could help but it requried a different way of interacting with openstack as the consuming applicaiton
and it somewhat limits you in terems for your network/volumn config unless you modify them after teh fact you have rretrived a reserved instance form nodepool.
On Thu, Mar 25, 2021 at 8:38 PM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-03-25 14:47:09 +0000 (+0000), Sean Mooney wrote:
nova does not support create a base vm and then doing a local live migration or restore for memory snapshots to create another vm.
this approch likely has several security implciations that would not be accpeatable in a multi tenant enviornment.
we have disucssed this type of vm creation in the past and determined that it is not a valid implematnion of spawn. a virt driver that precreate vms or copys an existing instance can be faster but that virt driver is not considered a compliant implementation.
so in short there is no way to achive this today in a compliant openstack powered cloud.
[...]
The next best thing is basically what Nodepool[*] does: start new virtual machines ahead of time and keep them available in the tenant. This does of course mean you're occupying additional quota for whatever base "ready" capacity you've set for your various images/flavors, and that you need to be able to predict how many of what kinds of virtual machines you're going to need in advance.
Is it recommended to use nodepool in a production environment?
-- Jeremy Stanley
On 2021-04-19 16:31:24 +0530 (+0530), open infra wrote:
On Thu, Mar 25, 2021 at 8:38 PM Jeremy Stanley fungi@yuggoth.org wrote:
[...]
The next best thing is basically what Nodepool[*] does: start new virtual machines ahead of time and keep them available in the tenant. This does of course mean you're occupying additional quota for whatever base "ready" capacity you've set for your various images/flavors, and that you need to be able to predict how many of what kinds of virtual machines you're going to need in advance.
Is it recommended to use nodepool in a production environment?
I can't begin to guess what you mean by "in a production environment," but it forms the lifecycle management basis for our production CI/CD system (as it does for many other Zuul installations). In the case of the deployment I help run, it's continuously connected to over a dozen production clouds, both public and private.
But anyway, I didn't say "use Nodepool." I suggested you look at "what Nodepool does" as a model for starting server instances in advance within the tenants/projects which regularly require instant access to new virtual machines.
FWIW I had some excellent startup times in Fort Nebula due to using local storage backed by nvme drives. Once the cloud image is copied to the hypervisor, startup's of the vms were usually measured in seconds. Not sure if that fits the requirements, but sub 30 second startups were the norm. This time was including the actual connection from nodepool to the instance. So I imagine the local start time was even faster.
What is the requirement for startup times?
On Mon, Apr 19, 2021 at 9:31 AM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-04-19 16:31:24 +0530 (+0530), open infra wrote:
On Thu, Mar 25, 2021 at 8:38 PM Jeremy Stanley fungi@yuggoth.org
wrote: [...]
The next best thing is basically what Nodepool[*] does: start new virtual machines ahead of time and keep them available in the tenant. This does of course mean you're occupying additional quota for whatever base "ready" capacity you've set for your various images/flavors, and that you need to be able to predict how many of what kinds of virtual machines you're going to need in advance.
Is it recommended to use nodepool in a production environment?
I can't begin to guess what you mean by "in a production environment," but it forms the lifecycle management basis for our production CI/CD system (as it does for many other Zuul installations). In the case of the deployment I help run, it's continuously connected to over a dozen production clouds, both public and private.
But anyway, I didn't say "use Nodepool." I suggested you look at "what Nodepool does" as a model for starting server instances in advance within the tenants/projects which regularly require instant access to new virtual machines. -- Jeremy Stanley
https://grafana.opendev.org/d/BskTteEGk/nodepool-openedge?orgId=1&from=1...
I know nodepool keeps a number of instances up and available for use, but on FN it was usually tapped to the max so I am not sure this logic applies.
Anyways in my performance testing and tuning of Openstack to get it moving as fast as possible, my determination was local NVME storage fit the rapid fire use case the best. Next best was on shared NVME storage via ISCSI using cinder caching.
On Thu, Apr 22, 2021 at 10:06 AM Donny Davis donny@fortnebula.com wrote:
FWIW I had some excellent startup times in Fort Nebula due to using local storage backed by nvme drives. Once the cloud image is copied to the hypervisor, startup's of the vms were usually measured in seconds. Not sure if that fits the requirements, but sub 30 second startups were the norm. This time was including the actual connection from nodepool to the instance. So I imagine the local start time was even faster.
What is the requirement for startup times?
On Mon, Apr 19, 2021 at 9:31 AM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-04-19 16:31:24 +0530 (+0530), open infra wrote:
On Thu, Mar 25, 2021 at 8:38 PM Jeremy Stanley fungi@yuggoth.org
wrote: [...]
The next best thing is basically what Nodepool[*] does: start new virtual machines ahead of time and keep them available in the tenant. This does of course mean you're occupying additional quota for whatever base "ready" capacity you've set for your various images/flavors, and that you need to be able to predict how many of what kinds of virtual machines you're going to need in advance.
Is it recommended to use nodepool in a production environment?
I can't begin to guess what you mean by "in a production environment," but it forms the lifecycle management basis for our production CI/CD system (as it does for many other Zuul installations). In the case of the deployment I help run, it's continuously connected to over a dozen production clouds, both public and private.
But anyway, I didn't say "use Nodepool." I suggested you look at "what Nodepool does" as a model for starting server instances in advance within the tenants/projects which regularly require instant access to new virtual machines. -- Jeremy Stanley
-- ~/DonnyD C: 805 814 6800 "No mission too difficult. No sacrifice too great. Duty First"
On Thu, Apr 22, 2021 at 7:46 PM Donny Davis donny@fortnebula.com wrote:
https://grafana.opendev.org/d/BskTteEGk/nodepool-openedge?orgId=1&from=1...
I know nodepool keeps a number of instances up and available for use, but on FN it was usually tapped to the max so I am not sure this logic applies.
Anyways in my performance testing and tuning of Openstack to get it moving as fast as possible, my determination was local NVME storage fit the rapid fire use case the best. Next best was on shared NVME storage via ISCSI using cinder caching.
Local nvme means, utilization of worker node storage?
On Thu, Apr 22, 2021 at 10:06 AM Donny Davis donny@fortnebula.com wrote:
FWIW I had some excellent startup times in Fort Nebula due to using local storage backed by nvme drives. Once the cloud image is copied to the hypervisor, startup's of the vms were usually measured in seconds. Not sure if that fits the requirements, but sub 30 second startups were the norm. This time was including the actual connection from nodepool to the instance. So I imagine the local start time was even faster.
What is the requirement for startup times?
End-user supposed to access an application based on his/her choice, and it's a dedicated VM for the user. Based on the application end-user selection, VM should be available to the end-user along with required apps and provided hardware.
On Mon, Apr 19, 2021 at 9:31 AM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-04-19 16:31:24 +0530 (+0530), open infra wrote:
On Thu, Mar 25, 2021 at 8:38 PM Jeremy Stanley fungi@yuggoth.org
wrote: [...]
The next best thing is basically what Nodepool[*] does: start new virtual machines ahead of time and keep them available in the tenant. This does of course mean you're occupying additional quota for whatever base "ready" capacity you've set for your various images/flavors, and that you need to be able to predict how many of what kinds of virtual machines you're going to need in advance.
Is it recommended to use nodepool in a production environment?
I can't begin to guess what you mean by "in a production environment," but it forms the lifecycle management basis for our production CI/CD system (as it does for many other Zuul installations). In the case of the deployment I help run, it's continuously connected to over a dozen production clouds, both public and private.
But anyway, I didn't say "use Nodepool." I suggested you look at "what Nodepool does" as a model for starting server instances in advance within the tenants/projects which regularly require instant access to new virtual machines. -- Jeremy Stanley
-- ~/DonnyD C: 805 814 6800 "No mission too difficult. No sacrifice too great. Duty First"
-- ~/DonnyD C: 805 814 6800 "No mission too difficult. No sacrifice too great. Duty First"
On Thu, Apr 22, 2021 at 10:24 AM open infra openinfradn@gmail.com wrote:
On Thu, Apr 22, 2021 at 7:46 PM Donny Davis donny@fortnebula.com wrote:
https://grafana.opendev.org/d/BskTteEGk/nodepool-openedge?orgId=1&from=1...
I know nodepool keeps a number of instances up and available for use, but on FN it was usually tapped to the max so I am not sure this logic applies.
Anyways in my performance testing and tuning of Openstack to get it moving as fast as possible, my determination was local NVME storage fit the rapid fire use case the best. Next best was on shared NVME storage via ISCSI using cinder caching.
Local nvme means, utilization of worker node storage?
On Thu, Apr 22, 2021 at 10:06 AM Donny Davis donny@fortnebula.com wrote:
FWIW I had some excellent startup times in Fort Nebula due to using local storage backed by nvme drives. Once the cloud image is copied to the hypervisor, startup's of the vms were usually measured in seconds. Not sure if that fits the requirements, but sub 30 second startups were the norm. This time was including the actual connection from nodepool to the instance. So I imagine the local start time was even faster.
What is the requirement for startup times?
End-user supposed to access an application based on his/her choice, and it's a dedicated VM for the user. Based on the application end-user selection, VM should be available to the end-user along with required apps and provided hardware.
On Mon, Apr 19, 2021 at 9:31 AM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-04-19 16:31:24 +0530 (+0530), open infra wrote:
On Thu, Mar 25, 2021 at 8:38 PM Jeremy Stanley fungi@yuggoth.org
wrote: [...]
The next best thing is basically what Nodepool[*] does: start new virtual machines ahead of time and keep them available in the tenant. This does of course mean you're occupying additional quota for whatever base "ready" capacity you've set for your various images/flavors, and that you need to be able to predict how many of what kinds of virtual machines you're going to need in advance.
Is it recommended to use nodepool in a production environment?
I can't begin to guess what you mean by "in a production environment," but it forms the lifecycle management basis for our production CI/CD system (as it does for many other Zuul installations). In the case of the deployment I help run, it's continuously connected to over a dozen production clouds, both public and private.
But anyway, I didn't say "use Nodepool." I suggested you look at "what Nodepool does" as a model for starting server instances in advance within the tenants/projects which regularly require instant access to new virtual machines. -- Jeremy Stanley
-- ~/DonnyD C: 805 814 6800 "No mission too difficult. No sacrifice too great. Duty First"
-- ~/DonnyD C: 805 814 6800 "No mission too difficult. No sacrifice too great. Duty First"
Local nvme means, utilization of worker node storage?
Yes, the compute nodes were configured to just use the local storage (which is the default I do believe). The directory /var/lib/nova was mounted onto a dedicated NVME device.
End-user supposed to access an application based on his/her choice, and
it's a dedicated VM for the user.
Based on the application end-user selection, VM should be available to the
end-user along with required apps and provided hardware. This is really a two part answer. You need a portal or mechanism for end users to request a templated application stack (meaning it may take one or more machines to support the application) You also need a method to create cloud images with your applications pre-baked into them. I would also consider creating a mechanism to bake the cloud images with all of the application code and configuration (best you can) already contained in the cloud image. Disk Image Builder is some really great software that can handle the image creation part for you. The Openstack infra team does this on the daily to support the CI. They use DIB, so it's battle tested.
It is likely you would end up creating some portal or using a CMP (Cloud Management Platform) to handle end user requests. For instance End users want access to X application, the CMP would orchestrate wiring together all of the bits and pieces and then respond back with access information to the app. You can use Openstack's Heat to do the same thing depending on what you want the experience for end users to be.
At this point in Openstack's maturity, it can do just about anything you want it to. You just need to be specific in what you are asking your infrastructure to do and configure it for the use case.
Hope this helps and good luck.
Cheers ~DonnyD
Thanks Donny.
On Thu, Apr 22, 2021 at 8:15 PM Donny Davis donny@fortnebula.com wrote:
On Thu, Apr 22, 2021 at 10:24 AM open infra openinfradn@gmail.com wrote:
On Thu, Apr 22, 2021 at 7:46 PM Donny Davis donny@fortnebula.com wrote:
https://grafana.opendev.org/d/BskTteEGk/nodepool-openedge?orgId=1&from=1...
I know nodepool keeps a number of instances up and available for use, but on FN it was usually tapped to the max so I am not sure this logic applies.
Anyways in my performance testing and tuning of Openstack to get it moving as fast as possible, my determination was local NVME storage fit the rapid fire use case the best. Next best was on shared NVME storage via ISCSI using cinder caching.
Local nvme means, utilization of worker node storage?
On Thu, Apr 22, 2021 at 10:06 AM Donny Davis donny@fortnebula.com wrote:
FWIW I had some excellent startup times in Fort Nebula due to using local storage backed by nvme drives. Once the cloud image is copied to the hypervisor, startup's of the vms were usually measured in seconds. Not sure if that fits the requirements, but sub 30 second startups were the norm. This time was including the actual connection from nodepool to the instance. So I imagine the local start time was even faster.
What is the requirement for startup times?
End-user supposed to access an application based on his/her choice, and it's a dedicated VM for the user. Based on the application end-user selection, VM should be available to the end-user along with required apps and provided hardware.
On Mon, Apr 19, 2021 at 9:31 AM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-04-19 16:31:24 +0530 (+0530), open infra wrote:
On Thu, Mar 25, 2021 at 8:38 PM Jeremy Stanley fungi@yuggoth.org
wrote: [...]
> The next best thing is basically what Nodepool[*] does: start new > virtual machines ahead of time and keep them available in the > tenant. This does of course mean you're occupying additional quota > for whatever base "ready" capacity you've set for your various > images/flavors, and that you need to be able to predict how many of > what kinds of virtual machines you're going to need in advance. > > [*] https://zuul-ci.org/docs/nodepool/
Is it recommended to use nodepool in a production environment?
I can't begin to guess what you mean by "in a production environment," but it forms the lifecycle management basis for our production CI/CD system (as it does for many other Zuul installations). In the case of the deployment I help run, it's continuously connected to over a dozen production clouds, both public and private.
But anyway, I didn't say "use Nodepool." I suggested you look at "what Nodepool does" as a model for starting server instances in advance within the tenants/projects which regularly require instant access to new virtual machines. -- Jeremy Stanley
-- ~/DonnyD C: 805 814 6800 "No mission too difficult. No sacrifice too great. Duty First"
-- ~/DonnyD C: 805 814 6800 "No mission too difficult. No sacrifice too great. Duty First"
Local nvme means, utilization of worker node storage?
Yes, the compute nodes were configured to just use the local storage (which is the default I do believe). The directory /var/lib/nova was mounted onto a dedicated NVME device.
End-user supposed to access an application based on his/her choice, and
it's a dedicated VM for the user.
Based on the application end-user selection, VM should be available to
the end-user along with required apps and provided hardware. This is really a two part answer. You need a portal or mechanism for end users to request a templated application stack (meaning it may take one or more machines to support the application) You also need a method to create cloud images with your applications pre-baked into them. I would also consider creating a mechanism to bake the cloud images with all of the application code and configuration (best you can) already contained in the cloud image. Disk Image Builder is some really great software that can handle the image creation part for you. The Openstack infra team does this on the daily to support the CI. They use DIB, so it's battle tested.
It is likely you would end up creating some portal or using a CMP (Cloud Management Platform) to handle end user requests. For instance End users want access to X application, the CMP would orchestrate wiring together all of the bits and pieces and then respond back with access information to the app. You can use Openstack's Heat to do the same thing depending on what you want the experience for end users to be.
At this point in Openstack's maturity, it can do just about anything you want it to. You just need to be specific in what you are asking your infrastructure to do and configure it for the use case.
I was supposed to use StarlingX to manage Openstack's underlying infrastructure and storage to keep separately (from worker/compute nodes) but not sure how this affects the boot time of VMs. Trying to achieve low latency.
Hope this helps and good luck.
Thanks again!
Cheers ~DonnyD
On Thu, 22 Apr 2021 at 15:08, Donny Davis donny@fortnebula.com wrote:
FWIW I had some excellent startup times in Fort Nebula due to using local storage backed by nvme drives. Once the cloud image is copied to the hypervisor, startup's of the vms were usually measured in seconds.
Nova image pre-caching might be useful here: https://docs.openstack.org/nova/ussuri/admin/image-caching.html#image-pre-ca...
Cheers, Ollie
Not sure if that fits the requirements, but sub 30 second startups were the norm. This time was including the actual connection from nodepool to the instance. So I imagine the local start time was even faster.
What is the requirement for startup times?
On Mon, Apr 19, 2021 at 9:31 AM Jeremy Stanley fungi@yuggoth.org wrote:
On 2021-04-19 16:31:24 +0530 (+0530), open infra wrote:
On Thu, Mar 25, 2021 at 8:38 PM Jeremy Stanley fungi@yuggoth.org
wrote: [...]
The next best thing is basically what Nodepool[*] does: start new virtual machines ahead of time and keep them available in the tenant. This does of course mean you're occupying additional quota for whatever base "ready" capacity you've set for your various images/flavors, and that you need to be able to predict how many of what kinds of virtual machines you're going to need in advance.
Is it recommended to use nodepool in a production environment?
I can't begin to guess what you mean by "in a production environment," but it forms the lifecycle management basis for our production CI/CD system (as it does for many other Zuul installations). In the case of the deployment I help run, it's continuously connected to over a dozen production clouds, both public and private.
But anyway, I didn't say "use Nodepool." I suggested you look at "what Nodepool does" as a model for starting server instances in advance within the tenants/projects which regularly require instant access to new virtual machines. -- Jeremy Stanley
-- ~/DonnyD C: 805 814 6800 "No mission too difficult. No sacrifice too great. Duty First"
On Thu, Mar 25, 2021 at 8:17 PM Sean Mooney smooney@redhat.com wrote:
This is a demo of a third party extention that was never upstreamed.
nova does not support create a base vm and then doing a local live migration or restore for memory snapshots to create another vm.
I just need to understand the risk and impact here but not desperately
trying to use the technology. Let say there won't be multiple tenants, but different users supposed to access stateless VMs. Is it still secure?
this approch likely has several security implciations that would not be accpeatable in a multi tenant enviornment.
we have disucssed this type of vm creation in the past and determined that it is not a valid implematnion of spawn. a virt driver that precreate vms or copys an existing instance can be faster but that virt driver is not considered a compliant implementation.
so in short there is no way to achive this today in a compliant openstack powered cloud.
On 30/03/2021 13:23, open infra wrote:
On Thu, Mar 25, 2021 at 8:17 PM Sean Mooney <smooney@redhat.com mailto:smooney@redhat.com> wrote:
This is a demo of a third party extention that was never upstreamed. nova does not support create a base vm and then doing a local live migration or restore for memory snapshots to create another vm.I just need to understand the risk and impact here but not desperately trying to use the technology. Let say there won't be multiple tenants, but different users supposed to access stateless VMs. Is it still secure?
you will need to ask the third party vendor who forked openstack to produce it. in general i dont think cross projefct/teant shareing of stateless vm memory would be safe. we dont know why the image is loading into memory when it boots. within the same project it might be but upstream cant really say since we have not review that code. what i would be most worried about is any keys that might be loaded by cloud init or similar that would be differnt between instances. im skeptical that this is actually a generic solution that should be implemented in a cloud environment.
this approch likely has several security implciations that would not be accpeatable in a multi tenant enviornment. we have disucssed this type of vm creation in the past and determined that it is not a valid implematnion of spawn. a virt driver that precreate vms or copys an existing instance can be faster but that virt driver is not considered a compliant implementation. so in short there is no way to achive this today in a compliant openstack powered cloud.
You could look at the work of a former Nova PTL that tries to solve this problem: https://shakenfist.com/
Does anyone deploy Meghdwar with OpenStack? Seems it wasn't active for the last few years
participants (6)
-
Donny Davis
-
Jeremy Stanley
-
Mike Carden
-
Oliver Walsh
-
open infra
-
Sean Mooney