[placement][ptg] Allocation Partitioning
From the etherpad [1]:
* do we need this? * what is it? * who is going to drive it? As as I recall, allocation partitioning (distinct from resource provider partitioning) is a way of declaring that a set of allocations below to a specific service. This is useful as a way, for example, of counting instances spawned via nova. Right now it is possible to count vcpus, ram and disk but if something besides nova is making allocations using those resource classes how do we distinguish? Of course it's important to also ask "should we distinguish?". If there's a concept of unified limits does it matter whether a VCPU is consume by _this_ nova or something else if they are consumed by the same user? This functionality is closely tied to resource provider partitioning. In a complex placement scenario, where placement is managing multiple instance spawning tools, in multiple cloud-like things, it seems like both would be needed. The ongoing work to implement quota counting in placement [2] has a workaround for instances not being counted in placement, but the "more than one nova per placement" limitation has to be documented. How urgent is this? Is there anyone available to do the work this cycle? How damaging is it to punt to U? What details are missing in the above description? [1] https://etherpad.openstack.org/p/placement-ptg-train [2] https://review.openstack.org/#/q/topic:bp/count-quota-usage-from-placement -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
On 04/08/2019 10:25 AM, Chris Dent wrote:
From the etherpad [1]:
* do we need this? * what is it? * who is going to drive it?
As as I recall, allocation partitioning (distinct from resource provider partitioning) is a way of declaring that a set of allocations below to a specific service. This is useful as a way, for example, of counting instances spawned via nova. Right now it is possible to count vcpus, ram and disk but if something besides nova is making allocations using those resource classes how do we distinguish?
This is more "consumer type" than anything else. In order to satisfy usage requests for certain classes of quota (really, just "num instances"), we need to know which consumers in the consumers table are "Nova instances" and which are not. Obviously, all consumers are currently Nova instances in placement since (AFAIK) no other services have begun using placement to store allocations. And because placement doesn't have such concepts as "local deleted" or "soft deleted allocations" we could actually accurately count the number of instances using a single simple query: SELECT COUNT(DISTINCT a.consumer_id) FROM allocations AS a JOIN consumers AS c ON a.consumer_id = c.id WHERE c.project_id = $PROJECT_ID AND c.type_id = $NOVA_INSTANCE_TYPE_ID;
Of course it's important to also ask "should we distinguish?". If there's a concept of unified limits does it matter whether a VCPU is consume by _this_ nova or something else if they are consumed by the same user?
So this is referring to resource provider partition (source partition), not consumer type. For the problem of detecting whether an instance is in *this* Nova or another Nova deployment that uses the same placement service, we need a source partition identifier in the resource_providers table. Also keep in mind that Keystone's unified limits are still divisible by *region*, which would serve as a natural source partition identifier I suppose.
This functionality is closely tied to resource provider partitioning. In a complex placement scenario, where placement is managing multiple instance spawning tools, in multiple cloud-like things, it seems like both would be needed.
Yes, this. In order to get the number of Nova instances in a specific deployment of Nova (or region), the above query would instead look like this: SELECT COUNT(DISTINCT a.consumer_id) FROM allocations AS a JOIN consumers AS c ON a.consumer_id = c.id JOIN resource_providers AS rp ON a.resource_provider_id WHERE c.project_id = $PROJECT_ID AND c.type_id = $NOVA_INSTANCE_TYPE_ID AND rp.source_partition_id = $NOVA_REGION_ID;
The ongoing work to implement quota counting in placement [2] has a workaround for instances not being counted in placement, but the "more than one nova per placement" limitation has to be documented.
How urgent is this? Is there anyone available to do the work this cycle? How damaging is it to punt to U? What details are missing in the above description?
Not sure of the answers to this question. As with most features like this, 95% of the work seems to end up being in the upgrade path and not adding the feature itself. For both the consumer type and source partition identification, the upgrade path would entail setting default consumer type for all existing consumers and source partition identifiers for all existing resource providers. Best, -jay
[1] https://etherpad.openstack.org/p/placement-ptg-train [2] https://review.openstack.org/#/q/topic:bp/count-quota-usage-from-placement
On Tue, 9 Apr 2019, Jay Pipes wrote:
On 04/08/2019 10:25 AM, Chris Dent wrote:
First, I'm trying to integrate several different mental models here, which is probably why my terminology is a bit confused. There are similar but not concurrent models from Dan, Jay, Mel, and Matt (that I can recall) that we'll want to synchronized before we get too much further on this.
This is more "consumer type" than anything else. In order to satisfy usage requests for certain classes of quota (really, just "num instances"), we need to know which consumers in the consumers table are "Nova instances" and which are not.
This is one of the mental models: Identify a consumer as having a type: "this is a nova (any nova) instance". "consumer type", perhaps. That's slightly different from another one of the mental models which is: Identify a consumer as having a source: "this is from service X (which happens to be a nova)". "consumer source", perhaps. In the latter, at least some comments have involved there being multiples services (of the same _type_) in the same OpenStack. I don't know that this is a thing, yet, but it presumed to be something that is possible, so not something we want to prevent in the data model.
Of course it's important to also ask "should we distinguish?". If there's a concept of unified limits does it matter whether a VCPU is consume by _this_ nova or something else if they are consumed by the same user?
So this is referring to resource provider partition (source partition), not consumer type. For the problem of detecting whether an instance is in *this* Nova or another Nova deployment that uses the same placement service, we need a source partition identifier in the resource_providers table.
That's not what I was trying to get at with my comment. What I was wonder was: If there exists a quota for VCPU does it matter how those VCPU are being consumed in the entire quota domain? Does it matter if the consumption is via nova or some other compute provisioning tool? I'll try to address the multiple mental models for resource provider partitioning in the other thread. Identify and resolving the differences in mental models prior to the PTG is one of the main reasons for having these threads, so huge success so far. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
On 04/09/2019 10:02 AM, Chris Dent wrote:
On Tue, 9 Apr 2019, Jay Pipes wrote:
So this is referring to resource provider partition (source partition), not consumer type. For the problem of detecting whether an instance is in *this* Nova or another Nova deployment that uses the same placement service, we need a source partition identifier in the resource_providers table.
That's not what I was trying to get at with my comment. What I was wonder was:
If there exists a quota for VCPU does it matter how those VCPU are being consumed in the entire quota domain? Does it matter if the consumption is via nova or some other compute provisioning tool?
Ah, yes, I see now. I suppose what you're trying to determine is whether there is a viable use case for answering the question "given a single 'source partition' -- say, "my compute cloud region A" -- show me the usage of VCPU resources in my 'nova' compute hosts and my 'non-nova' compute hosts". I'm just not sure there's a viable use case there. Best, -jay
On Apr 10, 2019, at 4:41 AM, Jay Pipes <jaypipes@gmail.com> wrote:
I suppose what you're trying to determine is whether there is a viable use case for answering the question "given a single 'source partition' -- say, "my compute cloud region A" -- show me the usage of VCPU resources in my 'nova' compute hosts and my 'non-nova' compute hosts".
I'm just not sure there's a viable use case there.
I was thinking more of the use case where there are several systems (novas) creating resource providers in a single placement service, and one of the novas asks for resources for an instance. We would need something equivalent to a per-nova aggregate so that placement doesn't return RPs from one of the other novas. Of course, requiring an agg to be manually set would not be practical, so some sort of automatic sharding identifier would be required. I have no idea how to do this best, or even if this is a use case we want to be able to entertain. It might be sufficient to just say that separate systems that don't share resources should have their own placement. -- Ed Leafe
Obviously, all consumers are currently Nova instances in placement since (AFAIK) no other services have begun using placement to store allocations.
This is not the case, actually. In Nova, we can have consumers that are instances, as well as consumers that are migrations (related to instances, but distinct). In order to be able to distinguish between the two, a partitioning key (I'd prefer we call this consumer type too) is needed. The quota work needs this specifically, so that we can know the usage by instances separately from usage that is reserved or pinned by an ongoing migration.
How urgent is this? Is there anyone available to do the work this cycle? How damaging is it to punt to U? What details are missing in the above description?
I guess we defer this question to Melanie, but I think the ability to count most of the quota elements we want using placement instead of the current method hinges on the ability to distinguish these things. --Dan
On Tue, 16 Apr 2019, Dan Smith wrote:
Obviously, all consumers are currently Nova instances in placement since (AFAIK) no other services have begun using placement to store allocations.
This is not the case, actually. In Nova, we can have consumers that are instances, as well as consumers that are migrations (related to instances, but distinct). In order to be able to distinguish between the two, a partitioning key (I'd prefer we call this consumer type too) is needed. The quota work needs this specifically, so that we can know the usage by instances separately from usage that is reserved or pinned by an ongoing migration.
When you think of "consumer type" do you imagine it as an enumeration with members similar to "instance", "instance-migration", "reservation" where those types are the same in all placement deployments, including those which are being used by multiple clouds (or multiple novas). What I'm trying to make sure about is: Is this a concept that is fully distinct from resource provider partitioning/shards/whatever the right word is, solely associated a consumer and the allocations it has right now? If not, how so? Thanks. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
When you think of "consumer type" do you imagine it as an enumeration with members similar to "instance", "instance-migration", "reservation" where those types are the same in all placement deployments, including those which are being used by multiple clouds (or multiple novas).
I guess it seems more like it needs to be (loosely) an enum within a given shard. Like "I'm nova, my shard is $UUID and within that shard, my consumer types are instance, migration, etc". Might also be "I'm ironic, my shard is $UUID and within that shard, my consumer types are nova-instance, standalone-use, etc". I don't really want to codify the enum in placement, as I think it's just a key that the consuming service uses to correlate the "reason" for each allocation. Some (like nova) could be a static small enum, but for others it could need to be dynamic and scale per tenant or something like that.
What I'm trying to make sure about is: Is this a concept that is fully distinct from resource provider partitioning/shards/whatever the right word is, solely associated a consumer and the allocations it has right now? If not, how so?
I'd like it to be fully distinct. If it's not, then nova basically needs to define consumers as "instance-$nova_distinct_deployment_uuid" instead of "instance", effectively using the consumer type as both a consumer type and a shard key. --Dan
On Tue, 16 Apr 2019 10:33:45 -0700, Dan Smith <dms@danplanet.com> wrote:
When you think of "consumer type" do you imagine it as an enumeration with members similar to "instance", "instance-migration", "reservation" where those types are the same in all placement deployments, including those which are being used by multiple clouds (or multiple novas).
I guess it seems more like it needs to be (loosely) an enum within a given shard. Like "I'm nova, my shard is $UUID and within that shard, my consumer types are instance, migration, etc". Might also be "I'm ironic, my shard is $UUID and within that shard, my consumer types are nova-instance, standalone-use, etc".
I don't really want to codify the enum in placement, as I think it's just a key that the consuming service uses to correlate the "reason" for each allocation. Some (like nova) could be a static small enum, but for others it could need to be dynamic and scale per tenant or something like that.
What I'm trying to make sure about is: Is this a concept that is fully distinct from resource provider partitioning/shards/whatever the right word is, solely associated a consumer and the allocations it has right now? If not, how so?
I'd like it to be fully distinct. If it's not, then nova basically needs to define consumers as "instance-$nova_distinct_deployment_uuid" instead of "instance", effectively using the consumer type as both a consumer type and a shard key.
+1 to all of this. -melanie
On 04/16/2019 01:38 PM, melanie witt wrote:
On Tue, 16 Apr 2019 10:33:45 -0700, Dan Smith <dms@danplanet.com> wrote:
When you think of "consumer type" do you imagine it as an enumeration with members similar to "instance", "instance-migration", "reservation" where those types are the same in all placement deployments, including those which are being used by multiple clouds (or multiple novas).
I guess it seems more like it needs to be (loosely) an enum within a given shard. Like "I'm nova, my shard is $UUID and within that shard, my consumer types are instance, migration, etc". Might also be "I'm ironic, my shard is $UUID and within that shard, my consumer types are nova-instance, standalone-use, etc".
I don't really want to codify the enum in placement, as I think it's just a key that the consuming service uses to correlate the "reason" for each allocation. Some (like nova) could be a static small enum, but for others it could need to be dynamic and scale per tenant or something like that.
What I'm trying to make sure about is: Is this a concept that is fully distinct from resource provider partitioning/shards/whatever the right word is, solely associated a consumer and the allocations it has right now? If not, how so?
I'd like it to be fully distinct. If it's not, then nova basically needs to define consumers as "instance-$nova_distinct_deployment_uuid" instead of "instance", effectively using the consumer type as both a consumer type and a shard key.
+1 to all of this.
-melanie
Yeah, +1 from me as well. Great point about the migrations being separate consumer type than instance. Best, -jay
On Tue, 16 Apr 2019, Jay Pipes wrote:
On 04/16/2019 01:38 PM, melanie witt wrote:
On Tue, 16 Apr 2019 10:33:45 -0700, Dan Smith <dms@danplanet.com> wrote:
I guess it seems more like it needs to be (loosely) an enum within a given shard. Like "I'm nova, my shard is $UUID and within that shard, my consumer types are instance, migration, etc". Might also be "I'm ironic, my shard is $UUID and within that shard, my consumer types are nova-instance, standalone-use, etc".
I don't really want to codify the enum in placement, as I think it's just a key that the consuming service uses to correlate the "reason" for each allocation. Some (like nova) could be a static small enum, but for others it could need to be dynamic and scale per tenant or something like that.
[snip]
+1 to all of this.
Yeah, +1 from me as well. Great point about the migrations being separate consumer type than instance.
For reference, there's also some discussion in IRC [1] with me, Dan, Jay. The summary, if I'm understanding correctly, is something like this: The thing we want is a "consumer type", it is used to distinguish one type of a consumer (and thus group of allocations representing a "workload") from another. For example "these allocations are for an instance", "these are for a migration". Different services will need to be in charge of the distinctions they want to make. We can't predict all those services and distinctions, therefore placement should not be authoritative on types, each service is. In any given deployment, collaborating services (like nova, neutron, ironic, cinder) would need to agree on how they are not overlapping. That's out of scope for the placement side of things, but not irrelevant generally.
From an implementation standpoint, if/when we do this that makes consumer type a string associated with a consumer. To get overly detailed for this level of discussion: Since there will be a lot of duplication we should probably normalize it out to its own table.
When a /usages query is made, the consumer type could be optionally expressed, limiting results. Are multiple types something that needs to be expressed in one query? Not yet discussed. There's been no discussion yet about how the consumer type would be expressed when an allocation is written. Nor whether it will be required and what happens with existing consumers that don't have it. And most importantly, no volunteers for the work. Interested? Ideally, if we want to do this in Train, we'd have someone owning this before the PTG. Thanks. [1] http://eavesdrop.openstack.org/irclogs/%23openstack-placement/%23openstack-p... -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
When a /usages query is made, the consumer type could be optionally expressed, limiting results. Are multiple types something that needs to be expressed in one query? Not yet discussed.
I expect that Nova's quota stuff will want to be able to express multiples, FWIW. However, I think I can say for sure that the usage result will be the most useful for quota behavior if it returns usages grouped by type (whether or not a subset of types were in a filter query). --Dan
On Tue, 16 Apr 2019 12:26:37 -0700, Dan Smith <dms@danplanet.com> wrote:
When a /usages query is made, the consumer type could be optionally expressed, limiting results. Are multiple types something that needs to be expressed in one query? Not yet discussed.
I expect that Nova's quota stuff will want to be able to express multiples, FWIW. However, I think I can say for sure that the usage result will be the most useful for quota behavior if it returns usages grouped by type (whether or not a subset of types were in a filter query).
Yes, would be helpful to be able to get multiple consumer types in a single query. I agree that we would need to be able to sort through the different consumer types on the response side. For example, we'll want to be able to count quota usage as max('instance' usage, 'migration' usage) during a resize. And if a multiple consumer types filtered query returned a roll-up of all usages without grouping by consumer type, we would not be able to do the max thing we want to do without multiple separate queries to placement. Note that this reminds me that today we have to do two separate queries to placement, one filtered by project_id and one filtered by project_id and user_id in order to roll-up usage for a project and usage for a project user, for similar reasons. -melanie
On Tue, 16 Apr 2019, Dan Smith wrote:
When a /usages query is made, the consumer type could be optionally expressed, limiting results. Are multiple types something that needs to be expressed in one query? Not yet discussed.
I expect that Nova's quota stuff will want to be able to express multiples, FWIW. However, I think I can say for sure that the usage result will be the most useful for quota behavior if it returns usages grouped by type (whether or not a subset of types were in a filter query).
I find myself far more excited by modifying the query parameters than modifying the response structure. But if it makes life a bunch easier then I guess my excitement is not the important bit. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
On Tue, 16 Apr 2019 20:12:36 +0100 (BST), Chris Dent <cdent+os@anticdent.org> wrote:
On Tue, 16 Apr 2019, Jay Pipes wrote:
On 04/16/2019 01:38 PM, melanie witt wrote:
On Tue, 16 Apr 2019 10:33:45 -0700, Dan Smith <dms@danplanet.com> wrote:
I guess it seems more like it needs to be (loosely) an enum within a given shard. Like "I'm nova, my shard is $UUID and within that shard, my consumer types are instance, migration, etc". Might also be "I'm ironic, my shard is $UUID and within that shard, my consumer types are nova-instance, standalone-use, etc".
I don't really want to codify the enum in placement, as I think it's just a key that the consuming service uses to correlate the "reason" for each allocation. Some (like nova) could be a static small enum, but for others it could need to be dynamic and scale per tenant or something like that.
[snip]
+1 to all of this.
Yeah, +1 from me as well. Great point about the migrations being separate consumer type than instance.
For reference, there's also some discussion in IRC [1] with me, Dan, Jay.
The summary, if I'm understanding correctly, is something like this:
The thing we want is a "consumer type", it is used to distinguish one type of a consumer (and thus group of allocations representing a "workload") from another. For example "these allocations are for an instance", "these are for a migration".
Different services will need to be in charge of the distinctions they want to make. We can't predict all those services and distinctions, therefore placement should not be authoritative on types, each service is.
In any given deployment, collaborating services (like nova, neutron, ironic, cinder) would need to agree on how they are not overlapping. That's out of scope for the placement side of things, but not irrelevant generally.
I must admit that this last paragraph makes the idea of codifying the consumer types in placement more appealing. Guess it's a trade-off on where the effort goes. Projects collaborating to avoid overlap vs a heavier process and implementation for placement to provide the consumer types. -melanie
On Tue, 16 Apr 2019, Chris Dent wrote:
And most importantly, no volunteers for the work. Interested?
Ideally, if we want to do this in Train, we'd have someone owning this before the PTG.
I've made a story for this: https://storyboard.openstack.org/#!/story/2005473 The first task on it is to create a spec. If you want to do that, assign yourself. As stated in the contributing docs [1] not all features will require a spec, but as things one has several undecided features and a fair few alternatives as spec seems reasonable. [1] https://docs.openstack.org/placement/latest/contributor/contributing.html#ne... -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
On Tue, 16 Apr 2019 09:13:47 -0700, Dan Smith <dms@danplanet.com> wrote:
Obviously, all consumers are currently Nova instances in placement since (AFAIK) no other services have begun using placement to store allocations.
This is not the case, actually. In Nova, we can have consumers that are instances, as well as consumers that are migrations (related to instances, but distinct). In order to be able to distinguish between the two, a partitioning key (I'd prefer we call this consumer type too) is needed. The quota work needs this specifically, so that we can know the usage by instances separately from usage that is reserved or pinned by an ongoing migration.
How urgent is this? Is there anyone available to do the work this cycle? How damaging is it to punt to U? What details are missing in the above description?
I guess we defer this question to Melanie, but I think the ability to count most of the quota elements we want using placement instead of the current method hinges on the ability to distinguish these things.
The latest incarnation of the quota usage from placement work [1] has counting from placement as opt-in, based on a few reasons explained in the commit message. And the spec will be amended when/if there is confirmation the patches will land in this form. Given the opt-in nature of the quota stuff ^, it could be tempting to punt it. But my biased opinion is that it would be nice to get working on it so we can make counting quota usage from placement our default behavior sooner than later. Consumer types would enable us to count instances from placement, I believe. Current code [1] has to count instances from instance mappings in the API database. Consumer types would also enable us to implement the quota usage behavior we want during a resize, to take max('instance' usage, 'migration' usage). Finally, resource provider partitioning would allow quota usage from placement to be correct in an environment with multiple nova deployments sharing a single placement deployment. If we are able to fill in those gaps, we would be in a good position to make counting quota usage from placement our default behavior in nova. -melanie [1] https://review.openstack.org/#/c/638073/26
On 4/16/2019 4:03 PM, melanie witt wrote:
Consumer types would enable us to count instances from placement, I believe.
I don't see how. The GET /usages API doesn't return anything until there are allocations, and there aren't allocations until the scheduler picks a host and creates the allocations, and that might fail resulting in NoValidHost and a server in ERROR status but that doesn't mean there aren't any instances. Changing that would be a break in behavior for counting instances. -- Thanks, Matt
On Tue, 16 Apr 2019 19:13:05 -0500, Matt Riedemann <mriedemos@gmail.com> wrote:
On 4/16/2019 4:03 PM, melanie witt wrote:
Consumer types would enable us to count instances from placement, I believe.
I don't see how. The GET /usages API doesn't return anything until there are allocations, and there aren't allocations until the scheduler picks a host and creates the allocations, and that might fail resulting in NoValidHost and a server in ERROR status but that doesn't mean there aren't any instances. Changing that would be a break in behavior for counting instances.
True, but now that you mention it, it seems the current proposed patch for counting cores and ram from placement results in a similar break in behavior. Today, an ERROR instance that has never been scheduled will consume cores and ram quota usage, but would not consume any in the same scenario when we count cores and ram from placement because of lack of allocations. -melanie
On 4/16/2019 7:22 PM, melanie witt wrote:
Today, an ERROR instance that has never been scheduled will consume cores and ram quota usage
Hmm, I wonder if that was also a regression in Pike with counting quotas. Because that usage is not real and arguably shouldn't count against the tenant. -- Thanks, Matt
On Tue, 16 Apr 2019 19:32:57 -0500, Matt Riedemann <mriedemos@gmail.com> wrote:
On 4/16/2019 7:22 PM, melanie witt wrote:
Today, an ERROR instance that has never been scheduled will consume cores and ram quota usage
Hmm, I wonder if that was also a regression in Pike with counting quotas. Because that usage is not real and arguably shouldn't count against the tenant.
I don't think so, based on hazy anecdotal experience and a look at the code which does a quota reservation before scheduling and doesn't appear to rollback quota in any failure case in conductor: https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L313 I still have my Ocata devstack up, so I'll do a test and reply with the results for Ocata, Train, and Train + counting from placement. -melanie
On Tue, 16 Apr 2019 17:46:15 -0700, Melanie Witt <melwittt@gmail.com> wrote:
On Tue, 16 Apr 2019 19:32:57 -0500, Matt Riedemann <mriedemos@gmail.com> wrote:
On 4/16/2019 7:22 PM, melanie witt wrote:
Today, an ERROR instance that has never been scheduled will consume cores and ram quota usage
Hmm, I wonder if that was also a regression in Pike with counting quotas. Because that usage is not real and arguably shouldn't count against the tenant.
I don't think so, based on hazy anecdotal experience and a look at the code which does a quota reservation before scheduling and doesn't appear to rollback quota in any failure case in conductor:
https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L313
I still have my Ocata devstack up, so I'll do a test and reply with the results for Ocata, Train, and Train + counting from placement.
Here are the results: * Ocata behavior, unscheduled ERROR instance consumes quota usage for instances, cores, and ram: http://paste.openstack.org/show/749406 * Train behavior, unscheduled ERROR instance consumes quota usage for instances, cores, and ram: http://paste.openstack.org/show/749407 * Train behavior + counting from placement, consumes quota usage for instances (API database) but not for cores and ram (placement allocations): http://paste.openstack.org/show/749408 -melanie
On Wed, Apr 17, 2019 at 3:22 AM melanie witt <melwittt@gmail.com> wrote:
On Tue, 16 Apr 2019 17:46:15 -0700, Melanie Witt <melwittt@gmail.com> wrote:
On Tue, 16 Apr 2019 19:32:57 -0500, Matt Riedemann <mriedemos@gmail.com> wrote:
On 4/16/2019 7:22 PM, melanie witt wrote:
Today, an ERROR instance that has never been scheduled will consume cores and ram quota usage
Hmm, I wonder if that was also a regression in Pike with counting quotas. Because that usage is not real and arguably shouldn't count against the tenant.
I don't think so, based on hazy anecdotal experience and a look at the code which does a quota reservation before scheduling and doesn't appear to rollback quota in any failure case in conductor:
https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L313
I still have my Ocata devstack up, so I'll do a test and reply with the results for Ocata, Train, and Train + counting from placement.
Here are the results:
* Ocata behavior, unscheduled ERROR instance consumes quota usage for instances, cores, and ram: http://paste.openstack.org/show/749406
* Train behavior, unscheduled ERROR instance consumes quota usage for instances, cores, and ram: http://paste.openstack.org/show/749407
* Train behavior + counting from placement, consumes quota usage for instances (API database) but not for cores and ram (placement allocations): http://paste.openstack.org/show/749408
I guess this would be the expected/needed behaviour right? That is we count the instance but don't count the RAM and cores. Maybe we should fix this for the stable branches also in the current way of counting. -- Regards, Surya.
On 4/16/2019 4:03 PM, melanie witt wrote:
Consumer types would also enable us to implement the quota usage behavior we want during a resize, to take max('instance' usage, 'migration' usage).
To be clear, and you did the testing on this recently, but this is how quota usage worked pre-pike counting quotas in nova, correct? And that counting behavior was changed in Pike because we now only count the new flavor resources for vcpus/ram while the server is in VERIFY_RESIZE status, even if it was a resize down - which could prevent the user from reverting the resize if they would go over-quota during revert because we didn't track quota properly during a resize. So "behavior we want" meaning, "get us back to pre-pike behavior" but at the same time no one has really complained about this slight wrinkle since it happened in pike (I think it dawned on me last week). Meaning, it's low priority, yeah? -- Thanks, Matt
On Tue, 16 Apr 2019 19:18:03 -0500, Matt Riedemann <mriedemos@gmail.com> wrote:
On 4/16/2019 4:03 PM, melanie witt wrote:
Consumer types would also enable us to implement the quota usage behavior we want during a resize, to take max('instance' usage, 'migration' usage).
To be clear, and you did the testing on this recently, but this is how quota usage worked pre-pike counting quotas in nova, correct? And that counting behavior was changed in Pike because we now only count the new flavor resources for vcpus/ram while the server is in VERIFY_RESIZE status, even if it was a resize down - which could prevent the user from reverting the resize if they would go over-quota during revert because we didn't track quota properly during a resize.
Yes, that is how quota usage worked pre-Pike. And yes, the counting behavior in Pike means that we only count the vcpus, memory_mb, and disk_gb attributes that are set on the instance. Since those attributes are updated to the new flavor on the instance at finish_resize time (before confirm/revert), we get only the new flavor count, regardless of whether it was an upsize or a downsize. To do it in a way that reserves room for a revert of a downsize, we would have to defer saving the instance attributes until after a resize confirm, only if it's a downsize. I'm not sure whether it's worth the complexity.
So "behavior we want" meaning, "get us back to pre-pike behavior" but at the same time no one has really complained about this slight wrinkle since it happened in pike (I think it dawned on me last week). Meaning, it's low priority, yeah?
Yes, back to pre-Pike behavior is the ideal state, I think. But what I was saying is getting to pre-Pike behavior from the current counting from placement proposal is a significant improvement. The current counting from placement proposal necessarily results in "doubled" allocations for upsizes and downsizes. If we're interested in making counting from placement the default in the future, getting away from "doubled" allocations seems important. For completeness, here are the results of the tests I did with Ocata and Train devstacks: * Ocata behavior, uses max of old flavor and new flavor until confirmed/reverted: http://paste.openstack.org/show/749221 * Train behavior, uses new flavor until confirmed/reverted: http://paste.openstack.org/show/749266 * Train behavior + counting from placement, uses old flavor + new flavor until confirmed/reverted: http://paste.openstack.org/show/749267 -melanie
On Tue, Apr 16, 2019 at 11:05 PM melanie witt <melwittt@gmail.com> wrote:
On Tue, 16 Apr 2019 09:13:47 -0700, Dan Smith <dms@danplanet.com> wrote:
Obviously, all consumers are currently Nova instances in placement since (AFAIK) no other services have begun using placement to store allocations.
This is not the case, actually. In Nova, we can have consumers that are instances, as well as consumers that are migrations (related to instances, but distinct). In order to be able to distinguish between the two, a partitioning key (I'd prefer we call this consumer type too) is needed. The quota work needs this specifically, so that we can know the usage by instances separately from usage that is reserved or pinned by an ongoing migration.
How urgent is this? Is there anyone available to do the work this cycle? How damaging is it to punt to U? What details are missing in the above description?
I guess we defer this question to Melanie, but I think the ability to count most of the quota elements we want using placement instead of the current method hinges on the ability to distinguish these things.
The latest incarnation of the quota usage from placement work [1] has counting from placement as opt-in, based on a few reasons explained in the commit message. And the spec will be amended when/if there is confirmation the patches will land in this form.
Given the opt-in nature of the quota stuff ^, it could be tempting to punt it. But my biased opinion is that it would be nice to get working on it so we can make counting quota usage from placement our default behavior sooner than later.
+1
Consumer types would enable us to count instances from placement, I believe. Current code [1] has to count instances from instance mappings in the API database. Consumer types would also enable us to implement the quota usage behavior we want during a resize, to take max('instance' usage, 'migration' usage).
Finally, resource provider partitioning would allow quota usage from placement to be correct in an environment with multiple nova deployments sharing a single placement deployment.
If we are able to fill in those gaps, we would be in a good position to make counting quota usage from placement our default behavior in nova.
I agree that while we have started the quota counting from placement as an opt-in it would be good to get the missing pieces in to make it whole asap. I can offer help if needed for the pieces. -- Regards, Surya.
On Mon, 8 Apr 2019, Chris Dent wrote:
As as I recall, allocation partitioning (distinct from resource provider partitioning) is a way of declaring that a set of allocations below to a specific service. This is useful as a way, for example, of counting instances spawned via nova. Right now it is possible to count vcpus, ram and disk but if something besides nova is making allocations using those resource classes how do we distinguish?
[snip]
This functionality is closely tied to resource provider partitioning. In a complex placement scenario, where placement is managing multiple instance spawning tools, in multiple cloud-like things, it seems like both would be needed.
Above discussed at http://lists.openstack.org/pipermail/openstack-discuss/2019-April/004720.htm... Both these threads haven't really gone anywhere concrete, yet. Since I know they've expressed some interest in partitioning in various ways I've pinged Dan and Mel to join in when they get the chance. If any operators or people from the Edge or NFV communities want to chime in here, that would be excellent. If we don't hear anything before the PTG it will be difficult to prioritize work related to partitioning in the Train cycle. Thanks. -- Chris Dent ٩◔̯◔۶ https://anticdent.org/ freenode: cdent tw: @anticdent
participants (7)
-
Chris Dent
-
Dan Smith
-
Ed Leafe
-
Jay Pipes
-
Matt Riedemann
-
melanie witt
-
Surya Seetharaman