[nova][keystone] project tags in context for scheduling
Hi nova and keystone devs, We have a use case where we want to schedule a bunch of projects to specific compute nodes only. The aggregate_multitenancy_isolation isn’t viable because in some cases we will want thousands of projects to go to some hardware and it isn’t manageable/scaleable to do this in nova and aggregates. (Maybe it is and I’m being silly?) The best way I can think of doing this is to tag the keystone projects (or possibly set a custom property) and then write a custom nova scheduler filter to use the tag/property. The only issue I have is that tags/properties aren’t available to nova in it’s RequestContext. Can you think of a better way or a way that would work now? If this does in fact sound like a good way forward does authtoken middleware send this data downwards so nova could consume? If it does I assume it would be then as simple as adding these to the nova RequestContext? Thanks, Sam
On 2/26/2019 11:53 PM, Sam Morrison wrote:
We have a use case where we want to schedule a bunch of projects to specific compute nodes only. The aggregate_multitenancy_isolation isn’t viable because in some cases we will want thousands of projects to go to some hardware and it isn’t manageable/scaleable to do this in nova and aggregates. (Maybe it is and I’m being silly?)
Is the issue because of this? https://bugs.launchpad.net/nova/+bug/1802111 Or just in general. Because https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#t... fixes that problem, but is only available since Rocky. Also, I can't find it now but there was a public cloud workgroup bug in launchpad at one point where it was asking that the AggregateMultiTenancyIsolation filter work on keystone domains rather than a list of projects, so if those projects were all in the same domain you'd just specify the domain in the aggregate metadata than the thousands of projects which is your scaling issue. Tobias might remember that bug. -- Thanks, Matt
On 2/27/19 8:25 AM, Matt Riedemann wrote:
On 2/26/2019 11:53 PM, Sam Morrison wrote:
We have a use case where we want to schedule a bunch of projects to specific compute nodes only. The aggregate_multitenancy_isolation isn’t viable because in some cases we will want thousands of projects to go to some hardware and it isn’t manageable/scaleable to do this in nova and aggregates. (Maybe it is and I’m being silly?)
Is the issue because of this?
https://bugs.launchpad.net/nova/+bug/1802111
Or just in general. Because https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#t... fixes that problem, but is only available since Rocky.
Also, I can't find it now but there was a public cloud workgroup bug in launchpad at one point where it was asking that the AggregateMultiTenancyIsolation filter work on keystone domains rather than a list of projects, so if those projects were all in the same domain you'd just specify the domain in the aggregate metadata than the thousands of projects which is your scaling issue. Tobias might remember that bug.
I can't find this either, but the working group does have a couple generic domain support bugs, but they aren't very specific to this issue. I think Matt brings up an interesting point about domain support. Currently, it's pretty limited to keystone. Not a lot of other services rely on domain-scoped tokens for anything [0]. As far as context and middleware goes, keystonemiddleware validates tokens, sets headers, and oslo.context has the ability to convert those request headers [1] to attributes of the context object from loading the request environment [2]. Long story short, if the context is getting handled property and if you have access to it, you can pull the project_id off the context object and query more information about it from keystone directly (also assuming you have keystoneclient handy). [0] https://docs.openstack.org/keystone/latest/contributor/services.html#authori... [1] http://git.openstack.org/cgit/openstack/oslo.context/tree/oslo_context/conte... [2] http://git.openstack.org/cgit/openstack/oslo.context/tree/oslo_context/conte...
On 28 Feb 2019, at 1:25 am, Matt Riedemann <mriedemos@gmail.com> wrote:
On 2/26/2019 11:53 PM, Sam Morrison wrote:
We have a use case where we want to schedule a bunch of projects to specific compute nodes only. The aggregate_multitenancy_isolation isn’t viable because in some cases we will want thousands of projects to go to some hardware and it isn’t manageable/scaleable to do this in nova and aggregates. (Maybe it is and I’m being silly?)
Is the issue because of this?
https://bugs.launchpad.net/nova/+bug/1802111 <https://bugs.launchpad.net/nova/+bug/1802111> Or just in general. Because https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#t... fixes that problem, but is only available since Rocky.
Yeah essentially it is however it would be nice to manage this in keystone where is’t all in one place but this I think would work. Just upgraded to Queens so not far off too!
Also, I can't find it now but there was a public cloud workgroup bug in launchpad at one point where it was asking that the AggregateMultiTenancyIsolation filter work on keystone domains rather than a list of projects, so if those projects were all in the same domain you'd just specify the domain in the aggregate metadata than the thousands of projects which is your scaling issue. Tobias might remember that bug.
We actually already have a domain scheduler filter [1] but we have multiple levels of projects that intersects. Eg. We use domains to seperate our Australian and NZ projects. But we also have needs to schedule our projects based on their funding source. Thanks, Sam [1] https://github.com/NeCTAR-RC/nova/blob/nectar/queens/nova/cells/filters/rest... <https://github.com/NeCTAR-RC/nova/blob/nectar/queens/nova/cells/filters/restrict_domain.py>
--
Thanks,
Matt
participants (3)
-
Lance Bragstad
-
Matt Riedemann
-
Sam Morrison