[placement][nova][ptg] Summary: Consumer Types
We had a brief conversation in the placement room yesterday (Saturday May 5th) to confirm we were all on the same page with regard to consumer types. These provide a way to say that a set of allocations "is an instance" or "is a migration" and will help with quota accounting. We decided that since no one has stepped forward with a more complicated scenario, at this time, we will go with the simplest implementation, for now: * add a consumer types table that has a key and string (length to be determined, values controlled by clients) that represents a "type". For example (1, 'instance') * add a column on consumer table that takes one of those keys * create a new row in the types table only when a new type is created, don't worry about expiring them * provide an online migration to default existing consumers to 'instance' and treat unset types as 'instance' [1]. This probably needs some confirmation from mel and others that it is suitable. If not, please provide an alternative suggestion. * In a new microversion: allow queries to /usages to use a consumer type parameter to limit results to particular types and add 'consumer_type' key will be added to the body of an 'allocations' in both PUT and POST. * We did not discuss in the room, but the email thread [2] did: We may need to consider grouping /usages results by type but we could probably get by without changing that (and do multiple requests, sometimes). Surya, thank her very much, has volunteered to work on this and has started a spec at [3]. We have decided, again due to lack of expressed demand, to do any work (at this time) related to resource provider partitioning [4]. There's a pretty good idea on how to do this, but enough other stuff going on there's not time. Because we decided in that thread that any one resource provider can only be in one partition, there is also a very easy workaround: Run another placement server. It takes only a few minutes to set one up [5] This means that all of the client services of a single placement service need to coordinate on what consumer types they are using. (This was already true, but stated here for emphasis.) [1] I'm tempted to test how long a million or so rows of consumers would take to update. If it is short enough we may wish to break with the nova tradition of not doing data migrations in schema migrations (placement-manage db sync). But we didn't get a chance to discuss that in the room. [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/thread.htm... [3] https://review.opendev.org/#/c/654799/ [4] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004721.htm... [5] https://docs.openstack.org/placement/latest/install/from-pypi.html -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
On Mon, May 6, 2019 at 1:54 AM, Chris Dent <cdent+os@anticdent.org> wrote:
We had a brief conversation in the placement room yesterday (Saturday May 5th) to confirm we were all on the same page with regard to consumer types. These provide a way to say that a set of allocations "is an instance" or "is a migration" and will help with quota accounting.
We decided that since no one has stepped forward with a more complicated scenario, at this time, we will go with the simplest implementation, for now:
* add a consumer types table that has a key and string (length to be determined, values controlled by clients) that represents a "type". For example (1, 'instance')
* add a column on consumer table that takes one of those keys
* create a new row in the types table only when a new type is created, don't worry about expiring them
* provide an online migration to default existing consumers to 'instance' and treat unset types as 'instance' [1]. This probably needs some confirmation from mel and others that it is suitable. If not, please provide an alternative suggestion.
If there are ongoing migration then defaulting the consumer type to instance might be incorrect. However nova already has a mechanism to distingush between migration and instance consumer so nova won't break by this. Still nova might want to fix this placement data inconsistency. I guess the new placement microversion will allow to update the consumer type of an allocation. Cheers, gibi
* In a new microversion: allow queries to /usages to use a consumer type parameter to limit results to particular types and add 'consumer_type' key will be added to the body of an 'allocations' in both PUT and POST.
* We did not discuss in the room, but the email thread [2] did: We may need to consider grouping /usages results by type but we could probably get by without changing that (and do multiple requests, sometimes).
Surya, thank her very much, has volunteered to work on this and has started a spec at [3].
We have decided, again due to lack of expressed demand, to do any work (at this time) related to resource provider partitioning [4].
There's a pretty good idea on how to do this, but enough other stuff going on there's not time. Because we decided in that thread that any one resource provider can only be in one partition, there is also a very easy workaround: Run another placement server. It takes only a few minutes to set one up [5]
This means that all of the client services of a single placement service need to coordinate on what consumer types they are using. (This was already true, but stated here for emphasis.)
[1] I'm tempted to test how long a million or so rows of consumers would take to update. If it is short enough we may wish to break with the nova tradition of not doing data migrations in schema migrations (placement-manage db sync). But we didn't get a chance to discuss that in the room.
[2] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/thread.htm...
[4] http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004721.htm...
[5] https://docs.openstack.org/placement/latest/install/from-pypi.html
-- Chris Dent ٩◔̯◔۶ https://protect2.fireeye.com/url?k=b585a35f-e90ea863-b585e3c4-86ef624f95b6-8f691958e6e41ae2&u=https://anticdent.org/ freenode: cdent tw: @anticdent
If there are ongoing migration then defaulting the consumer type to instance might be incorrect.
Right, and you have to assume that there are some in progress. Only Nova has the ability to tell you which consumers are instances or migrations. If we did this before we split, we could have looked at the api db instance mappings to make the determination, but I think now we need to be told via the API which is which.
However nova already has a mechanism to distingush between migration and instance consumer so nova won't break by this.
This would mean placement just lies about what each consumer is, and an operator trying to make sense of an upgrade by dumping stuff with osc-placement won't be able to tell the difference. They might be inclined to delete what, to them, would look like a bunch of stale instance allocations.
Still nova might want to fix this placement data inconsistency. I guess the new placement microversion will allow to update the consumer type of an allocation.
Yeah, I think this has to be updated from Nova. I (and I imagine others) would like to avoid making the type field optional in the API. So maybe default the value to something like "incomplete" or "unknown" and then let nova correct this naturally for instances on host startup and migrations on complete/revert. Ideally nova will be one one of the users that wants to depend on the type string, so we want to use our knowledge of which is which to get existing allocations updated so we can depend on the type value later. --Dan
On Mon, 6 May 2019, Dan Smith wrote:
Still nova might want to fix this placement data inconsistency. I guess the new placement microversion will allow to update the consumer type of an allocation.
Yeah, I think this has to be updated from Nova. I (and I imagine others) would like to avoid making the type field optional in the API. So maybe default the value to something like "incomplete" or "unknown" and then let nova correct this naturally for instances on host startup and migrations on complete/revert. Ideally nova will be one one of the users that wants to depend on the type string, so we want to use our knowledge of which is which to get existing allocations updated so we can depend on the type value later.
Ah, okay, good. If something like "unknown" is workable I think that's much much better than defaulting to instance. Thanks. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
On 5/6/2019 10:49 AM, Chris Dent wrote:
Still nova might want to fix this placement data inconsistency. I guess the new placement microversion will allow to update the consumer type of an allocation.
Yeah, I think this has to be updated from Nova. I (and I imagine others) would like to avoid making the type field optional in the API. So maybe default the value to something like "incomplete" or "unknown" and then let nova correct this naturally for instances on host startup and migrations on complete/revert. Ideally nova will be one one of the users that wants to depend on the type string, so we want to use our knowledge of which is which to get existing allocations updated so we can depend on the type value later.
Ah, okay, good. If something like "unknown" is workable I think that's much much better than defaulting to instance. Thanks.
Yup I agree with everything said from a nova perspective. Our public cloud operators were just asking about leaked allocations and if there was tooling to report and clean that kind of stuff up. I explained we have the heal_allocations CLI but that's only going to create allocations for *instances* and only if those instances aren't deleted, but we don't have anything in nova that deals with detection and cleanup of leaked allocations, sort of like what this tooling does [1] but I think is different. So I was thinking about how we could write something in nova that reads the allocations from placement and checks to see if there is anything in there that doesn't match what we have for instances or migrations, i.e. the server was deleted but for whatever reason an allocation was leaked. To be able to determine what allocations are nova-specific today we'd have to guess based on the resource classes being used, namely VCPU and/or MEMORY_MB, but it of course gets more complicated once we start adding supported for nested allocations and such. So consumer type will help here, but we need it more than from the GET /usages API I think. If I were writing that kind of report/cleanup tool today, I'd probably want a GET /allocations API, but that might be too heavy (it would definitely require paging support I think). I could probably get by with using GET /resource_providers/{uuid}/allocations for each compute node we have in nova, but again that starts to get complicated with nested providers (what if the allocations are for VGPU?). Anyway, from a "it's better to have something than nothing at all" perspective it's probably easiest to just start with the easy thing and ask placement for allocations on all compute node providers and cross-check those consumers against what's in nova and if we find allocations that don't have a matching migration or instance we could optional delete them. [1] https://github.com/larsks/os-placement-tools -- Thanks, Matt
On Wed, 8 May 2019, Matt Riedemann wrote:
Yup I agree with everything said from a nova perspective. Our public cloud operators were just asking about leaked allocations and if there was tooling to report and clean that kind of stuff up. I explained we have the heal_allocations CLI but that's only going to create allocations for *instances* and only if those instances aren't deleted, but we don't have anything in nova that deals with detection and cleanup of leaked allocations, sort of like what this tooling does [1] but I think is different.
I continue to wish that we had (or could chose to make) functionality on the compute node, perhaps in response to a signal of some kind that said: performed a reset of inventory and allocations. So that in case of doubt we can use reality as the authoritative source of truth, not either of the nova or placement dbs. I'm not sure if that's feasible at this stage. I agree that healing allocations for instances that are known to exist is easy, but cleaning up allocations that got left behind is harder. It's simplified somewhat (from nova's perspective) in that there should only ever be one group of allocations (that is, a thing identified by a consumer uuid) for an instance. Right now, you can generate a list of known consumers of compute nodes by doing what you describe: traversing the allocations of each compute node provider. If we ever move to a state where the compute node doesn't provide resources (and thus will have no allocations) we won't be able to do that, and that's one of the reasons why I get resistant when we talk about moving VCPU to NUMA nodes in all cases. Which supports your assertion that maybe some day it would be nice to list allocations by type. Some day. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
On Mon, May 6, 2019 at 5:51 PM Chris Dent <cdent+os@anticdent.org> wrote:
On Mon, 6 May 2019, Dan Smith wrote:
Still nova might want to fix this placement data inconsistency. I guess the new placement microversion will allow to update the consumer type of an allocation.
Yeah, I think this has to be updated from Nova. I (and I imagine others) would like to avoid making the type field optional in the API. So maybe default the value to something like "incomplete" or "unknown" and then let nova correct this naturally for instances on host startup and migrations on complete/revert. Ideally nova will be one one of the users that wants to depend on the type string, so we want to use our knowledge of which is which to get existing allocations updated so we can depend on the type value later.
Ah, okay, good. If something like "unknown" is workable I think that's much much better than defaulting to instance. Thanks.
okay, the spec will take this approach then. Regards, Surya.
participants (5)
-
Balázs Gibizer
-
Chris Dent
-
Dan Smith
-
Matt Riedemann
-
Surya Seetharaman