Hello Openstack team, is it possible to customize the nova-scheduler via Python? If yes, how? Best regards Levon
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1]. Hope this helps, Stephen [1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow...
Best regards Levon
Hello Stephen, I am a student from Germany who is currently working on his bachelor thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example. Two cases should be solved! Case 1: A user A with a low priority uses a VM from Openstack with half performance of the available host. Then user B comes in with a high priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created. Case 2: A user A with a low priority uses a VM with half the performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users. These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user. I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start? My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train. My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-an-instance.... The nova scheduler https://docs.openstack.org/operations-guide/ops-customize-compute.html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details-of-specifi... to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made. What do you guys think about it? Does it seem like a good starting point for you or is it the wrong approach? I'm very happy to have found you!!! Thank you really much for your time! Best regards Levon -----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>; openstack@lists.openstack.org Betreff: Re: Customization of nova-scheduler On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1]. Hope this helps, Stephen [1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow...
Best regards Levon
Levon, In case you have not seen this yet, you might want to have a look at the work on preemptible instances: https://techblog.web.cern.ch/techblog/post/preemptible-instances/ This sounds like it is related and could be helpful. Cheers, Arne On 31.05.21 13:44, levonmelikbekjan@yahoo.de wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example.
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with half performance of the available host. Then user B comes in with a high priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created.
Case 2: A user A with a low priority uses a VM with half the performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-an-instance....
The nova scheduler https://docs.openstack.org/operations-guide/ops-customize-compute.html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details-of-specifi... to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting point for you or is it the wrong approach?
I'm very happy to have found you!!!
Thank you really much for your time!
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>; openstack@lists.openstack.org Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow...
Best regards Levon
On Mon, 2021-05-31 at 13:44 +0200, levonmelikbekjan@yahoo.de wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example.
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with half performance of the available host. Then user B comes in with a high priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created.
Case 2: A user A with a low priority uses a VM with half the performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user.
What you're describing is commonly referred to as "preemptible" or "spot" instances. This topic has a long, complicated history in nova and has yet to be implemented. Searching for "preemptible instances openstack" should yield you lots of discussion on the topic along with a few proof-of-concept approaches using external services or out-of-tree modifications to nova.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
As hinted above, this is likely to be a very difficult project given the fraught history of the idea. I don't want to dissuade you from this work but you should be aware of what you're getting into from the start. If you're serious about pursuing this, I suggest you first do some research on prior art. As noted above, there is lots of information on the internet about this. With this research done, you'll need to decide whether this is something you want to approach within nova itself, via out-of-tree extensions or via a third party project. If you're opting for integration with nova, then you'll need to think long and hard about how you would design such a system and start working on a spec (a design document) outlining your proposed solution. Details on how to write a spec are discussed at [1]. The only extension points nova offers today are scheduler filters and weighers so your options for an out-of-tree extension approach will be limited. A third party project will arguably be the easiest approach but you will be restricted to talking to nova's REST APIs which may limit the design somewhat. This Blazar spec [2] could give you some ideas on this approach (assuming it was never actually implemented, though it may well have been).
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-an-instance....
The nova scheduler https://docs.openstack.org/operations-guide/ops-customize-compute.html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details-of-specifi... to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting point for you or is it the wrong approach?
This could potentially work, but I suspect there will be serious performance implications with this, particularly at scale. Scheduler filters are historically used for simple things like "find me a group of hosts that have this metadata attribute I set on my image". Making API calls sounds like something that would take significant time and therefore slow down the schedule process. You'd also have to decide what your heuristic for deciding which VM(s) to delete would be, since there's nothing obvious in nova that you could use. You could use something as simple as filter extra specs or something as complicated as an external service. This should be lots to get you started. Once again, do make sure you're aware of what you're getting yourself into before you start. This could get complicated very quickly :) Cheers, Stephen
I'm very happy to have found you!!!
Thank you really much for your time!
[1] https://specs.openstack.org/openstack/nova-specs/readme.html [2] https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>; openstack@lists.openstack.org Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow...
Best regards Levon
Hello Stephen, thank you for your quick reply and your valuable information. I am well aware that this task will not be easy to accomplish. However, I like challenging tasks because you can learn a lot from them. Thanks for the warning, but I think you misunderstood me. It is not my intention to reserve ressources for anyone. Let me explain you my aim more detailed. Hosts will exist in our infrastructure that will belong to a user (owner). Each user will have an aggregate that will be assigned to his user object as an id in the "extra" attribute field. All of the compute nodes that will be owned by the owner are located within this host aggregate. If hosts from an aggregate that belong to someone are not in use, everyone else is allowed to use them (for example a user who does not have any servers in his possession). When the owner decides to create a VM and our cloud doesn't have enough resources, all servers will be deleted from his compute node based on the aggregate id which is located in his user object. Then the function "Launch instance" tries again to create his VM. You're right, the API requests will take some time. The only requests I will send are one-off: - Get user by name/id - Get server list - Get aggregate by id ... and maybe a few times the server delete call. Maybe it is possible to store aggregate information locally and access it with python?!?! Or maybe it is better to store all the host information directly in the user object without having always to call the aggregate API. Alternatively, hypervisors could be used to minimize the number of calls for deletion of servers. This would only be a onetime call at the very beginning of my python script to determine the amount of free and used resources. With hypervisors and the information of the required resources by the owner of an aggregate, I could delete specific servers without having to delete all of them. I like the feature with the aggregates very much, especially because it is possible to add new compute nodes at any time. Kind regards Levon -----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 18:21 An: levonmelikbekjan@yahoo.de; openstack@lists.openstack.org Betreff: Re: AW: Customization of nova-scheduler On Mon, 2021-05-31 at 13:44 +0200, levonmelikbekjan@yahoo.de wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example.
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with half performance of the available host. Then user B comes in with a high priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created.
Case 2: A user A with a low priority uses a VM with half the performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user.
What you're describing is commonly referred to as "preemptible" or "spot" instances. This topic has a long, complicated history in nova and has yet to be implemented. Searching for "preemptible instances openstack" should yield you lots of discussion on the topic along with a few proof-of-concept approaches using external services or out-of-tree modifications to nova.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
As hinted above, this is likely to be a very difficult project given the fraught history of the idea. I don't want to dissuade you from this work but you should be aware of what you're getting into from the start. If you're serious about pursuing this, I suggest you first do some research on prior art. As noted above, there is lots of information on the internet about this. With this research done, you'll need to decide whether this is something you want to approach within nova itself, via out-of-tree extensions or via a third party project. If you're opting for integration with nova, then you'll need to think long and hard about how you would design such a system and start working on a spec (a design document) outlining your proposed solution. Details on how to write a spec are discussed at [1]. The only extension points nova offers today are scheduler filters and weighers so your options for an out-of-tree extension approach will be limited. A third party project will arguably be the easiest approach but you will be restricted to talking to nova's REST APIs which may limit the design somewhat. This Blazar spec [2] could give you some ideas on this approach (assuming it was never actually implemented, though it may well have been).
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-an-instance....
The nova scheduler https://docs.openstack.org/operations-guide/ops-customize-compute.html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details-of-specifi... to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting point for you or is it the wrong approach?
This could potentially work, but I suspect there will be serious performance implications with this, particularly at scale. Scheduler filters are historically used for simple things like "find me a group of hosts that have this metadata attribute I set on my image". Making API calls sounds like something that would take significant time and therefore slow down the schedule process. You'd also have to decide what your heuristic for deciding which VM(s) to delete would be, since there's nothing obvious in nova that you could use. You could use something as simple as filter extra specs or something as complicated as an external service. This should be lots to get you started. Once again, do make sure you're aware of what you're getting yourself into before you start. This could get complicated very quickly :) Cheers, Stephen
I'm very happy to have found you!!!
Thank you really much for your time!
[1] https://specs.openstack.org/openstack/nova-specs/readme.html [2] https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>; openstack@lists.openstack.org Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-y our-own-filter
Best regards Levon
On Mon, 2021-05-31 at 13:44 +0200, levonmelikbekjan@yahoo.de wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example.
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with half performance of the available host. Then user B comes in with a high priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created.
Case 2: A user A with a low priority uses a VM with half the performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users.
one thing to keep in mind is that end users are not allow to know the capstity of the cloud in terms of number of host, the resouces on a host or what host there vm is placeed on. so as a user the conceph of "a low priority uses a VM from Openstack with half performance of the available host" is not something that you can express arctecurally in nova. flavor define the size of vms in absolute term i.e. 4GB of ram not relitve "50% of the host". we have a 3 laryer schuldeing prcoess that start with a query to the placment service for a set of quantitative resouce class and qualitative traits.
On Mon, 2021-05-31 at 17:21 +0100, Stephen Finucane wrote: that produces a set fo allcoation candiate against a serise of host that could fit the instance, we then filter those host useing python filters wich are boolean fucntion that either pass the host or reject it finally after filtering we weight the remaining hosts and selecet one to boot the vm. once you have completed a steph in this processs you can nolonger go to a previous step and you can never readd a host afteer it has been elimiated by placemnt or a filter to be considered again. as a result if you get the end of the avaiable hosts and there are none that can fix your vm we cannot delete a vm and start again without redoing all the work and possible facing with concurrent api requests. this is why this is a hard problem with out an external service that can rebalance exiting workloads and free up capsity.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user.
What you're describing is commonly referred to as "preemptible" or "spot" instances. This topic has a long, complicated history in nova and has yet to be implemented. Searching for "preemptible instances openstack" should yield you lots of discussion on the topic along with a few proof-of-concept approaches using external services or out-of-tree modifications to nova.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
As hinted above, this is likely to be a very difficult project given the fraught history of the idea. I don't want to dissuade you from this work but you should be aware of what you're getting into from the start. If you're serious about pursuing this, I suggest you first do some research on prior art. As noted above, there is lots of information on the internet about this. With this research done, you'll need to decide whether this is something you want to approach within nova itself, via out-of-tree extensions or via a third party project. If you're opting for integration with nova, then you'll need to think long and hard about how you would design such a system and start working on a spec (a design document) outlining your proposed solution. Details on how to write a spec are discussed at [1]. The only extension points nova offers today are scheduler filters and weighers so your options for an out-of-tree extension approach will be limited. A third party project will arguably be the easiest approach but you will be restricted to talking to nova's REST APIs which may limit the design somewhat. This Blazar spec [2] could give you some ideas on this approach (assuming it was never actually implemented, though it may well have been).
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-an-instance....
The nova scheduler https://docs.openstack.org/operations-guide/ops-customize-compute.html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details-of-specifi... to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting point for you or is it the wrong approach?
This could potentially work, but I suspect there will be serious performance implications with this, particularly at scale. Scheduler filters are historically used for simple things like "find me a group of hosts that have this metadata attribute I set on my image". Making API calls sounds like something that would take significant time and therefore slow down the schedule process. You'd also have to decide what your heuristic for deciding which VM(s) to delete would be, since there's nothing obvious in nova that you could use. You could use something as simple as filter extra specs or something as complicated as an external service. yes implementing preemption in the scheduler as filet was disccused in the passed and discounted for the performance implication stephen hinted at. in tree we currentlyt do not allow filter to make any api or db queires. that approach also will not work toady since you would have to rexecute the query to the placment service after deleting an instance when you run out of capacity and restart the filtering which a filter cannot do as i noted above.
the most recent spec in this area was https://review.opendev.org/c/openstack/nova-specs/+/438640 for the integrated approch and https://review.opendev.org/c/openstack/nova-specs/+/554212/12 which proposed adding a pending state for use with a standalone service https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype ther are a number of presentation on this form cern/stackhapc https://www.stackhpc.com/scientific-sig-at-the-dublin-ptg.html http://openstack-in-production.blogspot.com/2018/02/maximizing-resource-util... https://openlab.cern/sites/openlab.web.cern.ch/files/2018-07/Containers_on_B... https://indico.cern.ch/event/739089/sessions/282073/attachments/1689073/2717... the current state is rebuilding from cell0 is not support but the pending state was never added and the reaper service was not upstream. work in this are has now move the blazar project as stphen noted in [2] https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem... but is dont think it has made much progress. https://review.opendev.org/q/topic:%22preemptibles%22+(status:open%20OR%20st...) nova previously had a pluggable scheduler that would have allowed you to reimplent the scudler entirely from scratch but we removed that capability in the last year or two. at this point the only viable approach that will not take multiple upstream cycles to this is really to use an external service.
This should be lots to get you started. Once again, do make sure you're aware of what you're getting yourself into before you start. This could get complicated very quickly :)
yes anything other then adding the pending state to nova will be very complex due to placement interaction. you would really need to implement a fallback query mechanism in the scudler iteself. anything after the call to placement is already too late. you might be able to reuse consumer types to make some allocation preemtiblae and have a prefilter decide if an allocation should be a normal nova consumer or premtable consumer based on a flavor extra spec.https://docs.openstack.org/placement/train/specs/train/approved/2005473-supp... this would still require the pending state and an external reaper service to free the capsity to be clean but its a possible direction.
Cheers, Stephen
I'm very happy to have found you!!!
Thank you really much for your time!
[1] https://specs.openstack.org/openstack/nova-specs/readme.html [2] https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>; openstack@lists.openstack.org Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow...
Best regards Levon
Hi Sean, maybe this is the time to bring up again the discussion regarding preemptible instances support in Nova. Preemptible/Spot instances are available in all of the major public clouds to allow a better resource utilization. OpenStack private clouds suffer exactly from the same issue. There was a lot of work done in this area during the last 3 years. Most of the work is summarized by the blogs/presentations/cern-gitlab that you mentioned. CERN has been running this code in production since 1 year ago. It allows us to use the spare capacity in the compute nodes dedicated for specific services to run batch workloads. I heard that "ARDC Nectar Research Cloud" is also running it. I believe the work that was done is an excellent PoC. Also, to me this looks like it should be a Nova feature. Having an external project to support this functionality it's a huge overhead. cheers, Belmiro On Tue, Jun 1, 2021 at 11:03 PM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-05-31 at 13:44 +0200, levonmelikbekjan@yahoo.de wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with
half performance of the available host. Then user B comes in with a high
Case 2: A user A with a low priority uses a VM with half the
one thing to keep in mind is that end users are not allow to know the capstity of the cloud in terms of number of host, the resouces on a host or what host there vm is placeed on. so as a user the conceph of "a low priority uses a VM from Openstack with half performance of the available host" is not something that you can express arctecurally in nova. flavor define the size of vms in absolute term i.e. 4GB of ram not relitve "50% of the host". we have a 3 laryer schuldeing prcoess that start with a query to the
On Mon, 2021-05-31 at 17:21 +0100, Stephen Finucane wrote: thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example. priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created. performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users. placment service for a set of quantitative resouce class and qualitative traits. that produces a set fo allcoation candiate against a serise of host that could fit the instance, we then filter those host useing python filters wich are boolean fucntion that either pass the host or reject it finally after filtering we weight the remaining hosts and selecet one to boot the vm.
once you have completed a steph in this processs you can nolonger go to a previous step and you can never readd a host afteer it has been elimiated by placemnt or a filter to be considered again. as a result if you get the end of the avaiable hosts and there are none that can fix your vm we cannot delete a vm and start again without redoing all the work and possible facing with concurrent api requests. this is why this is a hard problem with out an external service that can rebalance exiting workloads and free up capsity.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user.
What you're describing is commonly referred to as "preemptible" or "spot" instances. This topic has a long, complicated history in nova and has yet to be implemented. Searching for "preemptible instances openstack" should yield you lots of discussion on the topic along with a few proof-of-concept approaches using external services or out-of-tree modifications to nova.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
As hinted above, this is likely to be a very difficult project given the fraught history of the idea. I don't want to dissuade you from this work but you should be aware of what you're getting into from the start. If you're serious about pursuing this, I suggest you first do some research on prior art. As noted above, there is lots of information on the internet about this. With this research done, you'll need to decide whether this is something you want to approach within nova itself, via out-of-tree extensions or via a third party project. If you're opting for integration with nova, then you'll need to think long and hard about how you would design such a system and start working on a spec (a design document) outlining your proposed solution. Details on how to write a spec are discussed at [1]. The only extension points nova offers today are scheduler filters and weighers so your options for an out-of-tree extension approach will be limited. A third party project will arguably be the easiest approach but you will be restricted to talking to nova's REST APIs which may limit the design somewhat. This Blazar spec [2] could give you some ideas on this approach (assuming it was never actually implemented, though it may well have been).
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-an-instance....
The nova scheduler
https://docs.openstack.org/operations-guide/ops-customize-compute.html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details-of-specifi... to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting
point for you or is it the wrong approach?
This could potentially work, but I suspect there will be serious performance implications with this, particularly at scale. Scheduler filters are historically used for simple things like "find me a group of hosts that have this metadata attribute I set on my image". Making API calls sounds like something that would take significant time and therefore slow down the schedule process. You'd also have to decide what your heuristic for deciding which VM(s) to delete would be, since there's nothing obvious in nova that you could use. You could use something as simple as filter extra specs or something as complicated as an external service. yes implementing preemption in the scheduler as filet was disccused in the passed and discounted for the performance implication stephen hinted at. in tree we currentlyt do not allow filter to make any api or db queires. that approach also will not work toady since you would have to rexecute the query to the placment service after deleting an instance when you run out of capacity and restart the filtering which a filter cannot do as i noted above.
the most recent spec in this area was https://review.opendev.org/c/openstack/nova-specs/+/438640 for the integrated approch and https://review.opendev.org/c/openstack/nova-specs/+/554212/12 which proposed adding a pending state for use with a standalone service
https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype
ther are a number of presentation on this form cern/stackhapc https://www.stackhpc.com/scientific-sig-at-the-dublin-ptg.html
http://openstack-in-production.blogspot.com/2018/02/maximizing-resource-util...
https://openlab.cern/sites/openlab.web.cern.ch/files/2018-07/Containers_on_B...
https://indico.cern.ch/event/739089/sessions/282073/attachments/1689073/2717...
the current state is rebuilding from cell0 is not support but the pending state was never added and the reaper service was not upstream.
work in this are has now move the blazar project as stphen noted in [2]
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem... but is dont think it has made much progress. https://review.opendev.org/q/topic:%22preemptibles%22+(status:open%20OR%20st...)
nova previously had a pluggable scheduler that would have allowed you to reimplent the scudler entirely from scratch but we removed that capability in the last year or two. at this point the only viable approach that will not take multiple upstream cycles to this is really to use an external service.
This should be lots to get you started. Once again, do make sure you're
aware of
what you're getting yourself into before you start. This could get complicated very quickly :)
yes anything other then adding the pending state to nova will be very complex due to placement interaction. you would really need to implement a fallback query mechanism in the scudler iteself. anything after the call to placement is already too late. you might be able to reuse consumer types to make some allocation preemtiblae and have a prefilter decide if an allocation should be a normal nova consumer or premtable consumer based on a flavor extra spec. https://docs.openstack.org/placement/train/specs/train/approved/2005473-supp... this would still require the pending state and an external reaper service to free the capsity to be clean but its a possible direction.
Cheers, Stephen
I'm very happy to have found you!!!
Thank you really much for your time!
[1] https://specs.openstack.org/openstack/nova-specs/readme.html [2]
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>;
openstack@lists.openstack.org
Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow...
Best regards Levon
Hi Sean,
maybe this is the time to bring up again the discussion regarding preemptible instances support in Nova. maybe realistically im not sure we have the capasity to do the detailed design required
On Wed, 2021-06-02 at 16:12 +0200, Belmiro Moreira wrote: this cycle but we could disucss it with an aim to having something ready for next cycle. i still think this is a valuable capablity which is partly why i brought this topic up with gibi this morning http://eavesdrop.openstack.org/irclogs/%23openstack-nova/latest.log.html#t20... his reply is here http://eavesdrop.openstack.org/irclogs/%23openstack-nova/latest.log.html#t20... i was exploring the question does the soon to be intoduced consumer types impact the desgin in any way. if unified limits was aware of consumer types and we had a placement:consumer_type=premtibale extra spec for example and we enhanced nova to use that we could adress some of the awkwardness in the current design where you have to have two project to do quota properly. effectively i think unified limits + consumer types shoudl probably be a prerequisite. we might want to revive the pending state also alhtough we now have rebuild form cell 0 i belvie so that may not be reuqied. if there is interst in this perhaps we should explore a subteam/popup team to pursue this again?
Preemptible/Spot instances are available in all of the major public clouds to allow a better resource utilization. OpenStack private clouds suffer exactly from the same issue.
There was a lot of work done in this area during the last 3 years.
Most of the work is summarized by the blogs/presentations/cern-gitlab that you mentioned.
CERN has been running this code in production since 1 year ago. It allows us to use the spare capacity in the compute nodes dedicated for specific services to run batch workloads.
yep i see utility in it for providing extra cloud capacity for ci also.
I heard that "ARDC Nectar Research Cloud" is also running it.
I believe the work that was done is an excellent PoC.
well since cern and netcar are potentialy running it already is that not an endorsement of an external agent approch :)
Also, to me this looks like it should be a Nova feature. Having an external project to support this functionality it's a huge overhead.
so we have been debaiting addign a new agent to nova for a while that would be responsible for runing some of the period healing type task. we were calling it nova audit as a place holder but it woudld basicaly do thing like arciving delete rows healing allocation ectra. the other logical approch would be to incoperate it into the nova conductor but im still not sold that it shoudl be in the nova tree. im not againt that either but perhaps a better apprcoh would be to create seperate repo that is a deliverable of nova based on the poc code and incubate it there. im really conviced that an external process is a huge overhead but also haveing to maintain the project release it ectra probably is. with that said i have always been a fan of the idea of having a common agent on a node that ran multiple services. e.g. a way to deploy nova api, nova conductor and nova scheduler as a singel binary to reduce the number of service you need to manage but i think that is a seperate topic.
cheers,
Belmiro
On Tue, Jun 1, 2021 at 11:03 PM Sean Mooney <smooney@redhat.com> wrote:
On Mon, 2021-05-31 at 13:44 +0200, levonmelikbekjan@yahoo.de wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with
half performance of the available host. Then user B comes in with a high
Case 2: A user A with a low priority uses a VM with half the
one thing to keep in mind is that end users are not allow to know the capstity of the cloud in terms of number of host, the resouces on a host or what host there vm is placeed on. so as a user the conceph of "a low priority uses a VM from Openstack with half performance of the available host" is not something that you can express arctecurally in nova. flavor define the size of vms in absolute term i.e. 4GB of ram not relitve "50% of the host". we have a 3 laryer schuldeing prcoess that start with a query to the
On Mon, 2021-05-31 at 17:21 +0100, Stephen Finucane wrote: thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example. priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created. performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users. placment service for a set of quantitative resouce class and qualitative traits. that produces a set fo allcoation candiate against a serise of host that could fit the instance, we then filter those host useing python filters wich are boolean fucntion that either pass the host or reject it finally after filtering we weight the remaining hosts and selecet one to boot the vm.
once you have completed a steph in this processs you can nolonger go to a previous step and you can never readd a host afteer it has been elimiated by placemnt or a filter to be considered again. as a result if you get the end of the avaiable hosts and there are none that can fix your vm we cannot delete a vm and start again without redoing all the work and possible facing with concurrent api requests. this is why this is a hard problem with out an external service that can rebalance exiting workloads and free up capsity.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user.
What you're describing is commonly referred to as "preemptible" or "spot" instances. This topic has a long, complicated history in nova and has yet to be implemented. Searching for "preemptible instances openstack" should yield you lots of discussion on the topic along with a few proof-of-concept approaches using external services or out-of-tree modifications to nova.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
As hinted above, this is likely to be a very difficult project given the fraught history of the idea. I don't want to dissuade you from this work but you should be aware of what you're getting into from the start. If you're serious about pursuing this, I suggest you first do some research on prior art. As noted above, there is lots of information on the internet about this. With this research done, you'll need to decide whether this is something you want to approach within nova itself, via out-of-tree extensions or via a third party project. If you're opting for integration with nova, then you'll need to think long and hard about how you would design such a system and start working on a spec (a design document) outlining your proposed solution. Details on how to write a spec are discussed at [1]. The only extension points nova offers today are scheduler filters and weighers so your options for an out-of-tree extension approach will be limited. A third party project will arguably be the easiest approach but you will be restricted to talking to nova's REST APIs which may limit the design somewhat. This Blazar spec [2] could give you some ideas on this approach (assuming it was never actually implemented, though it may well have been).
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-an-instance....
The nova scheduler
https://docs.openstack.org/operations-guide/ops-customize-compute.html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details-of-specifi... to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting
point for you or is it the wrong approach?
This could potentially work, but I suspect there will be serious performance implications with this, particularly at scale. Scheduler filters are historically used for simple things like "find me a group of hosts that have this metadata attribute I set on my image". Making API calls sounds like something that would take significant time and therefore slow down the schedule process. You'd also have to decide what your heuristic for deciding which VM(s) to delete would be, since there's nothing obvious in nova that you could use. You could use something as simple as filter extra specs or something as complicated as an external service. yes implementing preemption in the scheduler as filet was disccused in the passed and discounted for the performance implication stephen hinted at. in tree we currentlyt do not allow filter to make any api or db queires. that approach also will not work toady since you would have to rexecute the query to the placment service after deleting an instance when you run out of capacity and restart the filtering which a filter cannot do as i noted above.
the most recent spec in this area was https://review.opendev.org/c/openstack/nova-specs/+/438640 for the integrated approch and https://review.opendev.org/c/openstack/nova-specs/+/554212/12 which proposed adding a pending state for use with a standalone service
https://gitlab.cern.ch/ttsiouts/ReaperServicePrototype
ther are a number of presentation on this form cern/stackhapc https://www.stackhpc.com/scientific-sig-at-the-dublin-ptg.html
http://openstack-in-production.blogspot.com/2018/02/maximizing-resource-util...
https://openlab.cern/sites/openlab.web.cern.ch/files/2018-07/Containers_on_B...
https://indico.cern.ch/event/739089/sessions/282073/attachments/1689073/2717...
the current state is rebuilding from cell0 is not support but the pending state was never added and the reaper service was not upstream.
work in this are has now move the blazar project as stphen noted in [2]
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem... but is dont think it has made much progress. https://review.opendev.org/q/topic:%22preemptibles%22+(status:open%20OR%20st...)
nova previously had a pluggable scheduler that would have allowed you to reimplent the scudler entirely from scratch but we removed that capability in the last year or two. at this point the only viable approach that will not take multiple upstream cycles to this is really to use an external service.
This should be lots to get you started. Once again, do make sure you're
aware of
what you're getting yourself into before you start. This could get complicated very quickly :)
yes anything other then adding the pending state to nova will be very complex due to placement interaction. you would really need to implement a fallback query mechanism in the scudler iteself. anything after the call to placement is already too late. you might be able to reuse consumer types to make some allocation preemtiblae and have a prefilter decide if an allocation should be a normal nova consumer or premtable consumer based on a flavor extra spec. https://docs.openstack.org/placement/train/specs/train/approved/2005473-supp... this would still require the pending state and an external reaper service to free the capsity to be clean but its a possible direction.
Cheers, Stephen
I'm very happy to have found you!!!
Thank you really much for your time!
[1] https://specs.openstack.org/openstack/nova-specs/readme.html [2]
https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>;
openstack@lists.openstack.org
Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow...
Best regards Levon
Hi Stephen, I'm trying to customize my nova scheduler. However, if I change the nova.conf as it is written here https://docs.openstack.org/operations-guide/de/ops-customize-compute.html, then my python file cannot be found. How can I configure it correctly? Do you have any idea? My controller node is running with CENTOS 7. I couldn't install devstack because it is only supported for CENTOS 8 version. Best regards Levon -----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 18:21 An: levonmelikbekjan@yahoo.de; openstack@lists.openstack.org Betreff: Re: AW: Customization of nova-scheduler On Mon, 2021-05-31 at 13:44 +0200, levonmelikbekjan@yahoo.de wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example.
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with half performance of the available host. Then user B comes in with a high priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created.
Case 2: A user A with a low priority uses a VM with half the performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user.
What you're describing is commonly referred to as "preemptible" or "spot" instances. This topic has a long, complicated history in nova and has yet to be implemented. Searching for "preemptible instances openstack" should yield you lots of discussion on the topic along with a few proof-of-concept approaches using external services or out-of-tree modifications to nova.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
As hinted above, this is likely to be a very difficult project given the fraught history of the idea. I don't want to dissuade you from this work but you should be aware of what you're getting into from the start. If you're serious about pursuing this, I suggest you first do some research on prior art. As noted above, there is lots of information on the internet about this. With this research done, you'll need to decide whether this is something you want to approach within nova itself, via out-of-tree extensions or via a third party project. If you're opting for integration with nova, then you'll need to think long and hard about how you would design such a system and start working on a spec (a design document) outlining your proposed solution. Details on how to write a spec are discussed at [1]. The only extension points nova offers today are scheduler filters and weighers so your options for an out-of-tree extension approach will be limited. A third party project will arguably be the easiest approach but you will be restricted to talking to nova's REST APIs which may limit the design somewhat. This Blazar spec [2] could give you some ideas on this approach (assuming it was never actually implemented, though it may well have been).
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-an-instance....
The nova scheduler https://docs.openstack.org/operations-guide/ops-customize-compute.html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details-of-specifi... to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting point for you or is it the wrong approach?
This could potentially work, but I suspect there will be serious performance implications with this, particularly at scale. Scheduler filters are historically used for simple things like "find me a group of hosts that have this metadata attribute I set on my image". Making API calls sounds like something that would take significant time and therefore slow down the schedule process. You'd also have to decide what your heuristic for deciding which VM(s) to delete would be, since there's nothing obvious in nova that you could use. You could use something as simple as filter extra specs or something as complicated as an external service. This should be lots to get you started. Once again, do make sure you're aware of what you're getting yourself into before you start. This could get complicated very quickly :) Cheers, Stephen
I'm very happy to have found you!!!
Thank you really much for your time!
[1] https://specs.openstack.org/openstack/nova-specs/readme.html [2] https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>; openstack@lists.openstack.org Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-y our-own-filter
Best regards Levon
On Thu, 2021-06-10 at 17:21 +0200, levonmelikbekjan@yahoo.de wrote:
Hi Stephen,
I'm trying to customize my nova scheduler. However, if I change the nova.conf as it is written here https://docs.openstack.org/operations-guide/de/ops-customize-compute.html, then my python file cannot be found. How can I configure it correctly?
Do you have any idea?
My controller node is running with CENTOS 7. I couldn't install devstack because it is only supported for CENTOS 8 version.
That document is very old. You want [1], which documents how to do this properly. Hope this helps, Stephen [1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow...
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 18:21 An: levonmelikbekjan@yahoo.de; openstack@lists.openstack.org Betreff: Re: AW: Customization of nova-scheduler
On Mon, 2021-05-31 at 13:44 +0200, levonmelikbekjan@yahoo.de wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example.
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with half performance of the available host. Then user B comes in with a high priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created.
Case 2: A user A with a low priority uses a VM with half the performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user.
What you're describing is commonly referred to as "preemptible" or "spot" instances. This topic has a long, complicated history in nova and has yet to be implemented. Searching for "preemptible instances openstack" should yield you lots of discussion on the topic along with a few proof-of-concept approaches using external services or out-of-tree modifications to nova.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
As hinted above, this is likely to be a very difficult project given the fraught history of the idea. I don't want to dissuade you from this work but you should be aware of what you're getting into from the start. If you're serious about pursuing this, I suggest you first do some research on prior art. As noted above, there is lots of information on the internet about this. With this research done, you'll need to decide whether this is something you want to approach within nova itself, via out-of-tree extensions or via a third party project. If you're opting for integration with nova, then you'll need to think long and hard about how you would design such a system and start working on a spec (a design document) outlining your proposed solution. Details on how to write a spec are discussed at [1]. The only extension points nova offers today are scheduler filters and weighers so your options for an out-of-tree extension approach will be limited. A third party project will arguably be the easiest approach but you will be restricted to talking to nova's REST APIs which may limit the design somewhat. This Blazar spec [2] could give you some ideas on this approach (assuming it was never actually implemented, though it may well have been).
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-an-instance....
The nova scheduler https://docs.openstack.org/operations-guide/ops-customize-compute.html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details-of-specifi... to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting point for you or is it the wrong approach?
This could potentially work, but I suspect there will be serious performance implications with this, particularly at scale. Scheduler filters are historically used for simple things like "find me a group of hosts that have this metadata attribute I set on my image". Making API calls sounds like something that would take significant time and therefore slow down the schedule process. You'd also have to decide what your heuristic for deciding which VM(s) to delete would be, since there's nothing obvious in nova that you could use. You could use something as simple as filter extra specs or something as complicated as an external service.
This should be lots to get you started. Once again, do make sure you're aware of what you're getting yourself into before you start. This could get complicated very quickly :)
Cheers, Stephen
I'm very happy to have found you!!!
Thank you really much for your time!
[1] https://specs.openstack.org/openstack/nova-specs/readme.html [2] https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>; openstack@lists.openstack.org Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-y our-own-filter
Best regards Levon
On Tue, 2021-06-15 at 15:18 +0100, Stephen Finucane wrote:
On Thu, 2021-06-10 at 17:21 +0200, levonmelikbekjan@yahoo.de wrote:
Hi Stephen,
I'm trying to customize my nova scheduler. However, if I change the nova.conf as it is written here https://docs.openstack.org/operations-guide/de/ops-customize-compute.html , then my python file cannot be found. How can I configure it correctly?
Do you have any idea?
My controller node is running with CENTOS 7. I couldn't install devstack because it is only supported for CENTOS 8 version.
That document is very old. You want [1], which documents how to do this properly.
wwell that depend if they acatully want to write ther own filter yes but if they want to replace the scheduler with a new one we recently removed support for that right. previously we had several schduler implemtation like the caching scheduler and that old doc https://docs.openstack.org/operations-guide/de/ops-customize-compute.html descibes on how to replace the filter scheduler dirver with an new one. we deprecated it ussuri https://github.com/openstack/nova/commit/6a4cb24d39623930fd240e67d6501380345... and you finally removed the extention point in febuary https://github.com/openstack/nova/commit/5aeb3a387494c4559d183d1290db3c92a96... so from wallaby on you can nolonger write an alternitvie schduler implemenation out of tree without reverting that. so yes https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow... is how you customise schduling now but you cant customise the schduler itself out fo tree anymore.
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow...
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 18:21 An: levonmelikbekjan@yahoo.de; openstack@lists.openstack.org Betreff: Re: AW: Customization of nova-scheduler
On Mon, 2021-05-31 at 13:44 +0200, levonmelikbekjan@yahoo.de wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example.
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with half performance of the available host. Then user B comes in with a high priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created.
Case 2: A user A with a low priority uses a VM with half the performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user.
What you're describing is commonly referred to as "preemptible" or "spot" instances. This topic has a long, complicated history in nova and has yet to be implemented. Searching for "preemptible instances openstack" should yield you lots of discussion on the topic along with a few proof-of-concept approaches using external services or out-of-tree modifications to nova.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
As hinted above, this is likely to be a very difficult project given the fraught history of the idea. I don't want to dissuade you from this work but you should be aware of what you're getting into from the start. If you're serious about pursuing this, I suggest you first do some research on prior art. As noted above, there is lots of information on the internet about this. With this research done, you'll need to decide whether this is something you want to approach within nova itself, via out-of-tree extensions or via a third party project. If you're opting for integration with nova, then you'll need to think long and hard about how you would design such a system and start working on a spec (a design document) outlining your proposed solution. Details on how to write a spec are discussed at [1]. The only extension points nova offers today are scheduler filters and weighers so your options for an out-of-tree extension approach will be limited. A third party project will arguably be the easiest approach but you will be restricted to talking to nova's REST APIs which may limit the design somewhat. This Blazar spec [2] could give you some ideas on this approach (assuming it was never actually implemented, though it may well have been).
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-an-instance... .
The nova scheduler https://docs.openstack.org/operations-guide/ops-customize-compute.html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details-of-specifi... to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting point for you or is it the wrong approach?
This could potentially work, but I suspect there will be serious performance implications with this, particularly at scale. Scheduler filters are historically used for simple things like "find me a group of hosts that have this metadata attribute I set on my image". Making API calls sounds like something that would take significant time and therefore slow down the schedule process. You'd also have to decide what your heuristic for deciding which VM(s) to delete would be, since there's nothing obvious in nova that you could use. You could use something as simple as filter extra specs or something as complicated as an external service.
This should be lots to get you started. Once again, do make sure you're aware of what you're getting yourself into before you start. This could get complicated very quickly :)
Cheers, Stephen
I'm very happy to have found you!!!
Thank you really much for your time!
[1] https://specs.openstack.org/openstack/nova-specs/readme.html [2] https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar-preem...
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>; openstack@lists.openstack.org Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-y our-own-filter
Best regards Levon
Hi Sean, I am already done with my solution. Everything works as expected! :) Thank you for your support. You guys are great. Best regards Levon -----Ursprüngliche Nachricht----- Von: Sean Mooney <smooney@redhat.com> Gesendet: Dienstag, 15. Juni 2021 16:37 An: Stephen Finucane <stephenfin@redhat.com>; levonmelikbekjan@yahoo.de; openstack@lists.openstack.org Betreff: Re: AW: AW: Customization of nova-scheduler On Tue, 2021-06-15 at 15:18 +0100, Stephen Finucane wrote:
On Thu, 2021-06-10 at 17:21 +0200, levonmelikbekjan@yahoo.de wrote:
Hi Stephen,
I'm trying to customize my nova scheduler. However, if I change the nova.conf as it is written here https://docs.openstack.org/operations-guide/de/ops-customize-compute .html , then my python file cannot be found. How can I configure it correctly?
Do you have any idea?
My controller node is running with CENTOS 7. I couldn't install devstack because it is only supported for CENTOS 8 version.
That document is very old. You want [1], which documents how to do this properly.
wwell that depend if they acatully want to write ther own filter yes but if they want to replace the scheduler with a new one we recently removed support for that right. previously we had several schduler implemtation like the caching scheduler and that old doc https://docs.openstack.org/operations-guide/de/ops-customize-compute.html descibes on how to replace the filter scheduler dirver with an new one. we deprecated it ussuri https://github.com/openstack/nova/commit/6a4cb24d39623930fd240e67d6501380345... and you finally removed the extention point in febuary https://github.com/openstack/nova/commit/5aeb3a387494c4559d183d1290db3c92a96... so from wallaby on you can nolonger write an alternitvie schduler implemenation out of tree without reverting that. so yes https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow... is how you customise schduling now but you cant customise the schduler itself out fo tree anymore.
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-y our-own-filter
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 18:21 An: levonmelikbekjan@yahoo.de; openstack@lists.openstack.org Betreff: Re: AW: Customization of nova-scheduler
On Mon, 2021-05-31 at 13:44 +0200, levonmelikbekjan@yahoo.de wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example.
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with half performance of the available host. Then user B comes in with a high priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created.
Case 2: A user A with a low priority uses a VM with half the performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user.
What you're describing is commonly referred to as "preemptible" or "spot" instances. This topic has a long, complicated history in nova and has yet to be implemented. Searching for "preemptible instances openstack" should yield you lots of discussion on the topic along with a few proof-of-concept approaches using external services or out-of-tree modifications to nova.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
As hinted above, this is likely to be a very difficult project given the fraught history of the idea. I don't want to dissuade you from this work but you should be aware of what you're getting into from the start. If you're serious about pursuing this, I suggest you first do some research on prior art. As noted above, there is lots of information on the internet about this. With this research done, you'll need to decide whether this is something you want to approach within nova itself, via out-of-tree extensions or via a third party project. If you're opting for integration with nova, then you'll need to think long and hard about how you would design such a system and start working on a spec (a design document) outlining your proposed solution. Details on how to write a spec are discussed at [1]. The only extension points nova offers today are scheduler filters and weighers so your options for an out-of-tree extension approach will be limited. A third party project will arguably be the easiest approach but you will be restricted to talking to nova's REST APIs which may limit the design somewhat. This Blazar spec [2] could give you some ideas on this approach (assuming it was never actually implemented, though it may well have been).
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-a n-instance.png .
The nova scheduler https://docs.openstack.org/operations-guide/ops-customize-compute. html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details- of-specific-api-version-detail,list-servers-detail to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting point for you or is it the wrong approach?
This could potentially work, but I suspect there will be serious performance implications with this, particularly at scale. Scheduler filters are historically used for simple things like "find me a group of hosts that have this metadata attribute I set on my image". Making API calls sounds like something that would take significant time and therefore slow down the schedule process. You'd also have to decide what your heuristic for deciding which VM(s) to delete would be, since there's nothing obvious in nova that you could use. You could use something as simple as filter extra specs or something as complicated as an external service.
This should be lots to get you started. Once again, do make sure you're aware of what you're getting yourself into before you start. This could get complicated very quickly :)
Cheers, Stephen
I'm very happy to have found you!!!
Thank you really much for your time!
[1] https://specs.openstack.org/openstack/nova-specs/readme.html [2] https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blaz ar-preemptible-instances.html
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>; openstack@lists.openstack.org Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writi ng-y our-own-filter
Best regards Levon
Out of curiosity, how did you end up implementing your workflow? Through the scheduler or is the logic external to Openstack? On Tue, Jun 15, 2021 at 11:48 AM <levonmelikbekjan@yahoo.de> wrote:
Hi Sean,
I am already done with my solution. Everything works as expected! :)
Thank you for your support. You guys are great.
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Sean Mooney <smooney@redhat.com> Gesendet: Dienstag, 15. Juni 2021 16:37 An: Stephen Finucane <stephenfin@redhat.com>; levonmelikbekjan@yahoo.de; openstack@lists.openstack.org Betreff: Re: AW: AW: Customization of nova-scheduler
On Tue, 2021-06-15 at 15:18 +0100, Stephen Finucane wrote:
On Thu, 2021-06-10 at 17:21 +0200, levonmelikbekjan@yahoo.de wrote:
Hi Stephen,
I'm trying to customize my nova scheduler. However, if I change the nova.conf as it is written here https://docs.openstack.org/operations-guide/de/ops-customize-compute .html , then my python file cannot be found. How can I configure it correctly?
Do you have any idea?
My controller node is running with CENTOS 7. I couldn't install devstack because it is only supported for CENTOS 8 version.
That document is very old. You want [1], which documents how to do this properly.
wwell that depend if they acatully want to write ther own filter yes but if they want to replace the scheduler with a new one we recently removed support for that right. previously we had several schduler implemtation like the caching scheduler and that old doc https://docs.openstack.org/operations-guide/de/ops-customize-compute.html
descibes on how to replace the filter scheduler dirver with an new one. we deprecated it ussuri
https://github.com/openstack/nova/commit/6a4cb24d39623930fd240e67d6501380345...
and you finally removed the extention point in febuary
https://github.com/openstack/nova/commit/5aeb3a387494c4559d183d1290db3c92a96... so from wallaby on you can nolonger write an alternitvie schduler implemenation out of tree without reverting that.
so yes
https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow... is how you customise schduling now but you cant customise the schduler itself out fo tree anymore.
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-y our-own-filter
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 18:21 An: levonmelikbekjan@yahoo.de; openstack@lists.openstack.org Betreff: Re: AW: Customization of nova-scheduler
On Mon, 2021-05-31 at 13:44 +0200, levonmelikbekjan@yahoo.de wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example.
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with half performance of the available host. Then user B comes in with a high priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created.
Case 2: A user A with a low priority uses a VM with half the performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his
VM.
When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user.
What you're describing is commonly referred to as "preemptible" or "spot" instances. This topic has a long, complicated history in nova and has yet to be implemented. Searching for "preemptible instances openstack" should yield you lots of discussion on the topic along with a few proof-of-concept approaches using external services or out-of-tree modifications to nova.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
As hinted above, this is likely to be a very difficult project given the fraught history of the idea. I don't want to dissuade you from this work but you should be aware of what you're getting into from the start. If you're serious about pursuing this, I suggest you first do some research on prior art. As noted above, there is lots of information on the internet about this. With this research done, you'll need to decide whether this is something you want to approach within nova itself, via out-of-tree extensions or via a third party project. If you're opting for integration with nova, then you'll need to think long and hard about how you would design such a system and start working on a spec (a design document) outlining your proposed solution. Details on how to write a spec are discussed at [1]. The only extension points nova offers today are scheduler filters and weighers so your options for an out-of-tree extension approach will be limited. A third party project will arguably be the easiest approach but you will be restricted to talking to nova's REST APIs which may limit the design somewhat. This Blazar spec [2] could give you some ideas on this approach (assuming it was never actually implemented, though it may well have been).
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-a n-instance.png .
The nova scheduler https://docs.openstack.org/operations-guide/ops-customize-compute. html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details- of-specific-api-version-detail,list-servers-detail to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting point for you or is it the wrong approach?
This could potentially work, but I suspect there will be serious performance implications with this, particularly at scale. Scheduler filters are historically used for simple things like "find me a group of hosts that have this metadata attribute I set on my image". Making API calls sounds like something that would take significant time and therefore slow down the schedule process. You'd also have to decide what your heuristic for deciding which VM(s) to delete would be, since there's nothing obvious in nova that you could use. You could use something as simple as filter extra specs or something as complicated as an external service.
This should be lots to get you started. Once again, do make sure you're aware of what you're getting yourself into before you start. This could get complicated very quickly :)
Cheers, Stephen
I'm very happy to have found you!!!
Thank you really much for your time!
[1] https://specs.openstack.org/openstack/nova-specs/readme.html [2] https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blaz ar-preemptible-instances.html
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>; openstack@lists.openstack.org Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writi ng-y our-own-filter
Best regards Levon
I would like to explain you that, but now I am facing another issue. The quota info message appeared „Quota exceeded for cores: Requested 4, but already used 59 of 60 cores“. In my version Train is a path called /usr/lib/python2.7/site-packages/nova/openstack with a file called wsgi.py. To be more precise it is the __exit__ function in the ResourceExceptionHandler, which outputs this information message. My question is, where does this function get the information from that the limit of the VCPUs and possibly other resources has been exceeded? I mean the scheduler doesn’t even run. I can’t get it until now. Maybe I need to manipulate the nova-api rather then manipulating the nova-scheduler. Thank you very much and have a nice day! Best regards Levon Von: Laurent Dumont <laurentfdumont@gmail.com> Gesendet: Mittwoch, 16. Juni 2021 02:55 An: levonmelikbekjan@yahoo.de Cc: Sean Mooney <smooney@redhat.com>; openstack <openstack@lists.openstack.org> Betreff: Re: AW: AW: Customization of nova-scheduler Out of curiosity, how did you end up implementing your workflow? Through the scheduler or is the logic external to Openstack? On Tue, Jun 15, 2021 at 11:48 AM <levonmelikbekjan@yahoo.de <mailto:levonmelikbekjan@yahoo.de> > wrote: Hi Sean, I am already done with my solution. Everything works as expected! :) Thank you for your support. You guys are great. Best regards Levon -----Ursprüngliche Nachricht----- Von: Sean Mooney <smooney@redhat.com <mailto:smooney@redhat.com> > Gesendet: Dienstag, 15. Juni 2021 16:37 An: Stephen Finucane <stephenfin@redhat.com <mailto:stephenfin@redhat.com> >; levonmelikbekjan@yahoo.de <mailto:levonmelikbekjan@yahoo.de> ; openstack@lists.openstack.org <mailto:openstack@lists.openstack.org> Betreff: Re: AW: AW: Customization of nova-scheduler On Tue, 2021-06-15 at 15:18 +0100, Stephen Finucane wrote:
On Thu, 2021-06-10 at 17:21 +0200, levonmelikbekjan@yahoo.de <mailto:levonmelikbekjan@yahoo.de> wrote:
Hi Stephen,
I'm trying to customize my nova scheduler. However, if I change the nova.conf as it is written here https://docs.openstackorg/operations-guide/de/ops-customize-compute <https://docs.openstack.org/operations-guide/de/ops-customize-compute> .html , then my python file cannot be found. How can I configure it correctly?
Do you have any idea?
My controller node is running with CENTOS 7. I couldn't install devstack because it is only supported for CENTOS 8 version.
That document is very old. You want [1], which documents how to do this properly.
wwell that depend if they acatully want to write ther own filter yes but if they want to replace the scheduler with a new one we recently removed support for that right. previously we had several schduler implemtation like the caching scheduler and that old doc https://docs.openstack.org/operations-guide/de/ops-customize-compute.html descibes on how to replace the filter scheduler dirver with an new one. we deprecated it ussuri https://github.com/openstack/nova/commit/6a4cb24d39623930fd240e67d6501380345... and you finally removed the extention point in febuary https://github.com/openstack/nova/commit/5aeb3a387494c4559d183d1290db3c92a96... so from wallaby on you can nolonger write an alternitvie schduler implemenation out of tree without reverting that. so yes https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow... is how you customise schduling now but you cant customise the schduler itself out fo tree anymore.
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-y our-own-filter
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com <mailto:stephenfin@redhat.com> > Gesendet: Montag, 31. Mai 2021 18:21 An: levonmelikbekjan@yahoo.de <mailto:levonmelikbekjan@yahoo.de> ; openstack@lists.openstack.org <mailto:openstack@lists.openstack.org> Betreff: Re: AW: Customization of nova-scheduler
On Mon, 2021-05-31 at 13:44 +0200, levonmelikbekjan@yahoo.de <mailto:levonmelikbekjan@yahoo.de> wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with half performance of the available host. Then user B comes in with a high priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B The VM of user B is successfully created.
Case 2: A user A with a low priority uses a VM with half the performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user
What you're describing is commonly referred to as "preemptible" or "spot" instances. This topic has a long, complicated history in nova and has yet to be implemented. Searching for "preemptible instances openstack" should yield you lots of discussion on the topic along with a few proof-of-concept approaches using external services or out-of-tree modifications to nova.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
As hinted above, this is likely to be a very difficult project given the fraught history of the idea. I don't want to dissuade you from this work but you should be aware of what you're getting into from the start. If you're serious about pursuing this, I suggest you first do some research on prior art. As noted above, there is lots of information on the internet about this. With this research done, you'll need to decide whether this is something you want to approach within nova itself, via out-of-tree extensions or via a third party project. If you're opting for integration with nova, then you'll need to think long and hard about how you would design such a system and start working on a spec (a design document) outlining your proposed solution. Details on how to write a spec are discussed at [1]. The only extension points nova offers today are scheduler filters and weighers so your options for an out-of-tree extension approach will be limited. A third party project will arguably be the easiest approach but you will be restricted to talking to nova's REST APIs which may limit the design somewhat. This Blazar spec [2] could give you some ideas on this approach (assuming it was never actually implemented, though it may well have been).
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-a n-instance.png .
The nova scheduler https://docs.openstack.org/operations-guide/ops-customize-compute. html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details- of-specific-api-version-detail,list-servers-detail to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting point for you or is it the wrong approach?
This could potentially work, but I suspect there will be serious performance implications with this, particularly at scale. Scheduler filters are historically used for simple things like "find me a group of hosts that have this metadata attribute I set on my image". Making API calls sounds like something that would take significant time and therefore slow down the schedule process. You'd also have to decide what your heuristic for deciding which VM(s) to delete would be, since there's nothing obvious in nova that you could use. You could use something as simple as filter extra specs or something as complicated as an external service.
This should be lots to get you started. Once again, do make sure you're aware of what you're getting yourself into before you start. This could get complicated very quickly :)
Cheers, Stephen
I'm very happy to have found you!!!
Thank you really much for your time!
[1] https://specs.openstack.org/openstack/nova-specs/readme.html [2] https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blaz ar-preemptible-instances.html
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com <mailto:stephenfin@redhat.com> > Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de <mailto:levonmelikbekjan@yahoo.de> >; openstack@lists.openstack.org <mailto:openstack@lists.openstack.org> Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writi ng-y our-own-filter
Best regards Levon
On 6/16/21 11:22, levonmelikbekjan@yahoo.de wrote:
I would like to explain you that, but now I am facing another issue. The quota info message appeared „Quota exceeded for cores: Requested 4, but already used 59 of 60 cores“.
In my version Train is a path called /usr/lib/python2.7/site-packages/nova/openstack with a file called wsgi.py. To be more precise it is the __exit__ function in the ResourceExceptionHandler, which outputs this information message.
My question is, where does this function get the information from that the limit of the VCPUs and possibly other resources has been exceeded? I mean the scheduler doesn’t even run. I can’t get it until now. Maybe I need to manipulate the nova-api rather then manipulating the nova-scheduler.
I assume you mean you get this message when you try to create a server? The quota check first occurs in nova-api and it uses the --flavor requested to determine the requested cores and ram, then it counts non-deleted instances in the database's flavors cores and ram for the project and compares the totals against the quota limit set in the /os-quota-sets API [1] and if one hasn't been set there it uses the config option values under [quota] in nova.conf. Hope that helps, -melwitt [1] https://docs.openstack.org/api-ref/compute/#show-a-quota
Hi Stephen, I am already done with my solution. Everything works as expected! :) Thank you for your support. You guys are great. Best regards Levon -----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Dienstag, 15. Juni 2021 16:19 An: levonmelikbekjan@yahoo.de; openstack@lists.openstack.org Betreff: Re: AW: AW: Customization of nova-scheduler On Thu, 2021-06-10 at 17:21 +0200, levonmelikbekjan@yahoo.de wrote:
Hi Stephen,
I'm trying to customize my nova scheduler. However, if I change the novaconf as it is written here https://docs.openstack.org/operations-guide/de/ops-customize-compute.html, then my python file cannot be found. How can I configure it correctly?
Do you have any idea?
My controller node is running with CENTOS 7. I couldn't install devstack because it is only supported for CENTOS 8 version.
That document is very old. You want [1], which documents how to do this properly. Hope this helps, Stephen [1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing-your-ow...
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 18:21 An: levonmelikbekjan@yahoo.de; openstack@lists.openstack.org Betreff: Re: AW: Customization of nova-scheduler
On Mon, 2021-05-31 at 13:44 +0200, levonmelikbekjan@yahoo.de wrote:
Hello Stephen,
I am a student from Germany who is currently working on his bachelor thesis. My job is to build a cloud solution for my university with Openstack. The functionality should include the prioritization of users. So that you can imagine exactly how the whole thing should work, I would like to give you an example.
Two cases should be solved!
Case 1: A user A with a low priority uses a VM from Openstack with half performance of the available host. Then user B comes in with a high priority and needs the full performance of the host for his VM. When creating the VM of user B, the VM of user A should be deleted because there is not enough compute power for user B. The VM of user B is successfully created.
Case 2: A user A with a low priority uses a VM with half the performance of the available host, then user B comes in with a high priority and needs half of the performance of the host for his VM. When creating the VM of user B, user A should not be deleted, since enough computing power is available for both users.
These cases should work for unlimited users. In order to optimize the whole thing, I would like to write a function that precisely calculates all performance components to determine whether enough resources are available for the VM of the high priority user.
What you're describing is commonly referred to as "preemptible" or "spot" instances. This topic has a long, complicated history in nova and has yet to be implemented. Searching for "preemptible instances openstack" should yield you lots of discussion on the topic along with a few proof-of-concept approaches using external services or out-of-tree modifications to nova.
I’m new to Openstack, but I’ve already implemented cloud projects with Microsoft Azure and have solid programming skills. Can you give me a hint where and how I can start?
As hinted above, this is likely to be a very difficult project given the fraught history of the idea. I don't want to dissuade you from this work but you should be aware of what you're getting into from the start. If you're serious about pursuing this, I suggest you first do some research on prior art. As noted above, there is lots of information on the internet about this. With this research done, you'll need to decide whether this is something you want to approach within nova itself, via out-of-tree extensions or via a third party project. If you're opting for integration with nova, then you'll need to think long and hard about how you would design such a system and start working on a spec (a design document) outlining your proposed solution. Details on how to write a spec are discussed at [1]. The only extension points nova offers today are scheduler filters and weighers so your options for an out-of-tree extension approach will be limited. A third party project will arguably be the easiest approach but you will be restricted to talking to nova's REST APIs which may limit the design somewhat. This Blazar spec [2] could give you some ideas on this approach (assuming it was never actually implemented, though it may well have been).
My university gave me three compute hosts and one control host to implement this solution for the bachelor thesis. I’m currently setting up Openstack and all the services on the control host all by myself to understand all the functionality (sorry for not using Packstack) 😉. All my hosts have CentOS 7 and the minimum deployment which I configure is Train.
My idea is to work with nova schedulers, because they seem to be interesting for my case. I've found a whole infrastructure description of the provisioning of an instance in Openstack https://docs.openstack.org/operations-guide/de/_images/provision-an-instance....
The nova scheduler https://docs.openstack.org/operations-guide/ops-customize-compute.html is the first component, where it is possible to implement functions via Python and the Compute API https://docs.openstack.org/api-ref/compute/?expanded=show-details-of-specifi... to check for active VMs and probably delete them if needed before a successful request for an instantiation can be made.
What do you guys think about it? Does it seem like a good starting point for you or is it the wrong approach?
This could potentially work, but I suspect there will be serious performance implications with this, particularly at scale. Scheduler filters are historically used for simple things like "find me a group of hosts that have this metadata attribute I set on my image". Making API calls sounds like something that would take significant time and therefore slow down the schedule process. You'd also have to decide what your heuristic for deciding which VM(s) to delete would be, since there's nothing obvious in nova that you could use. You could use something as simple as filter extra specs or something as complicated as an external service.
This should be lots to get you started. Once again, do make sure you're aware of what you're getting yourself into before you start. This could get complicated very quickly :)
Cheers, Stephen
I'm very happy to have found you!!!
Thank you really much for your time!
[1] https://specs.openstack.org/openstack/nova-specs/readme.html [2] https://specs.openstack.org/openstack/blazar-specs/specs/ussuri/blazar -preemptible-instances.html
Best regards Levon
-----Ursprüngliche Nachricht----- Von: Stephen Finucane <stephenfin@redhat.com> Gesendet: Montag, 31. Mai 2021 12:34 An: Levon Melikbekjan <levonmelikbekjan@yahoo.de>; openstack@lists.openstack.org Betreff: Re: Customization of nova-scheduler
On Wed, 2021-05-26 at 22:46 +0200, Levon Melikbekjan wrote:
Hello Openstack team,
is it possible to customize the nova-scheduler via Python? If yes, how?
Yes, you can provide your own filters and weighers. This is documented at [1].
Hope this helps, Stephen
[1] https://docs.openstack.org/nova/latest/user/filter-scheduler#writing -y our-own-filter
Best regards Levon
participants (8)
-
Arne Wiebalck
-
Belmiro Moreira
-
Laurent Dumont
-
Levon Melikbekjan
-
levonmelikbekjan＠yahoo.de
-
melanie witt
-
Sean Mooney
-
Stephen Finucane