<div dir="auto"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Dec 2, 2016 5:21 PM, "Matt Riedemann" <<a href="mailto:mriedem@linux.vnet.ibm.com">mriedem@linux.vnet.ibm.com</a>> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="elided-text">On 12/2/2016 12:04 PM, Chris Dent wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Latest news on what's going on with resource providers and the<br>
placement API. I've made some adjustments in the structure of this<br>
since last time[0]. The new structure tries to put the stuff we need to<br>
talk about, including medium and long term planning, at the top and<br>
move the stuff that is summaries of what's going on on gerrit towards<br>
the bottom. I think we need to do this to enhance the opportunities for<br>
asynchronous resolution of some of the topics on our plates. If we<br>
keep waiting until the next meeting where we are all there at the same<br>
time, stuff will sit for too long.<br>
<br>
[0]<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2016-November/107982.html" rel="noreferrer" target="_blank">http://lists.openstack.org/pip<wbr>ermail/openstack-dev/2016-Nove<wbr>mber/107982.html</a><br>
<br>
<br>
# Things to Think About<br>
<br>
(Note that I'm frequently going to be wrong or at least incomplete<br>
about the things I say here, because I'm writing off the top of my<br>
head. Half the point of writing this is to get it correct by<br>
collaborative action. If you see something that is wrong, please<br>
shout out in a response. This section is for discussion of stuff that<br>
isn't yet being tracked well or has vague conflicts.)<br>
<br>
The general goal with placement for Ocata is to have both the nova<br>
scheduler and resource tracker talking to the API to usefully limit<br>
the number of hosts that the scheduler evaluates when selecting<br>
destinations. There are several segments of work coming together to<br>
make this possible, some of which are further along than others.<br>
<br>
## Update Client Side to Consider Aggregates<br>
<br>
When the scheduler requests a list of resource providers, that list<br>
ought to include compute nodes that are are associated, via<br>
aggregates, with any shared resource provides (such as shared disk)<br>
that can satisfy the resource requirements in the request.<br>
<br>
Meanwhile, when a compute node places a VM that uses shared disk the<br>
allocation of resources made by the resource tracker need to go to<br>
the right resource providers.<br>
<br>
This is a thing we know we need to do but is not something for which<br>
(as far as I know) we've articulated a clear plan or really started<br>
on.<br>
</blockquote>
<br></div>
I'm glad I'm not the only one that was wondering what's going on with the client side aggregates handling stuff.</blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">I have it all done locally. Will push tomorrow...</div><div dir="auto"><br></div><div dir="auto">Best,</div><div dir="auto">- jay</div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> I see the aggregates PUT/GET patches have merged but the resource tracker stuff hasn't started, at least that I'm aware of. I was looking into this a bit this week when writing up the Ocata priorities docs and needed to go back into the generic-resource-pools spec from Newton to dig into the notes on aggregates:<br>
<br>
<a href="https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/generic-resource-pools.html" rel="noreferrer" target="_blank">https://specs.openstack.org/op<wbr>enstack/nova-specs/specs/newto<wbr>n/implemented/generic-resource<wbr>-pools.html</a><br>
<br>
There is a lot of detail in there, which is good - even though we retrospected at the summit that we spent too much time on details in the specs in Newton, I guess in this case it might pay off. :)<br><br>
If I'm understanding correctly, a 'resource pool' in that spec when talking about aggregates is really a set of resource providers tied to an aggregate. So I could have 3 compute nodes A, B, C all using the same shared storage cluster X. So A, B, and C are in an aggregate for X and we have the resource providers for compute nodes A, B and C all related to that aggregate X in the placement service. How that ties back into the scheduler and resource tracker is a bit fuzzy to me at the moment, but if ^ is correct then I could probably figure the rest out by digging back into the spec details.<div class="quoted-text"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
## Update Scheduler to Request Limited Resource Providers<br>
<br>
The "Scheduler Filters in DB" spec[1] has merged along with its<br>
pair, "Filter Resource Providers by Request"[2], and the work has<br>
started[3].<br>
<br>
There are some things to consider as that work progresses:<br>
<br>
* The bit about aggregates in the previous section: the list of<br>
returned resource providers needs to include associated providers.<br>
</blockquote>
<br></div>
nit: I think you mean associated _aggregates_ here.<div class="quoted-text"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
To quote Mr. Pipes:<br>
<br>
we will only return resource providers to the scheduler that<br>
are compute nodes in Ocata. the resource providers that the<br>
placement service returns will either have the resources<br>
requested or will be associated with aggregates that have<br>
providers that match the requested resources.<br>
</blockquote>
<br></div>
An example might be useful here, but I'm sure there is probably already one in the generic resource pools spec linked above. I think it means:<br>
<br>
"have the resources requested"<br>
<br>
- means this is a resource provider that satisfies a request for some type of resource class, maybe DISK_GB.<div class="quoted-text"><br>
<br>
"or will be associated with aggregates that have providers that match the requested resources."<br>
<br></div>
- means there is a shared storage resource provider that's associated to an aggregate in the placement service and that aggregate is associated with some compute node resource providers? So in my example up above, does that mean we have a resource provider for the shared storage cluster, let's call it X, which is associated with aggregate (again, X), and compute nodes A, B and C are in that aggregate, and are resource providers themselves.<div class="quoted-text"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
* There is unresolved debate about the structure of the request being<br>
made to the API. Is it POST or a GET, does it have a body or use<br>
query strings? The plan is to resolve this discussion in the review<br>
of the code at [3].<br>
</blockquote>
<br></div>
I personally prefer the POST after reading about the differences between the two, and when reviewing the spec on this. I'm not crazy about the scheduler having to pass a giant json string as a query parameter to a GET request on the placement API, I'd rather do that with a request body.<div class="quoted-text"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
[1]<br>
<a href="http://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/resource-providers-scheduler-db-filters.html" rel="noreferrer" target="_blank">http://specs.openstack.org/ope<wbr>nstack/nova-specs/specs/ocata/<wbr>approved/resource-providers-<wbr>scheduler-db-filters.html</a><br>
<br>
[2]<br>
<a href="http://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/resource-providers-get-by-request.html" rel="noreferrer" target="_blank">http://specs.openstack.org/ope<wbr>nstack/nova-specs/specs/ocata/<wbr>approved/resource-providers-<wbr>get-by-request.html</a><br>
<br>
[3] <a href="https://review.openstack.org/#/c/386242/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/386242/</a><br>
<br>
## Docs<br>
<br>
In addition to needing an api-ref we also need a placement-dev.rst to<br>
go alongside the placement.rst. The -dev would mostly explain the how<br>
and the why of the placement API archicture, how the testing works,<br>
etc. That's mostly on me.<br>
<br>
## Placement Upgrade/Installation issues<br>
<br>
(This is a straight copy from the previous message)<br>
<br>
In his response[4] to this topic Matt R pointed out todos for this<br>
topic:<br>
<br>
* get the placement-api enabled by default in the various bits of<br>
ocata CI * ensure that microversions are being used on both sides of the<br>
placement API transactions (that's true in pending changes to<br>
both the API and the resource tracker)<br>
<br>
[4]<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2016-November/107177.html" rel="noreferrer" target="_blank">http://lists.openstack.org/pip<wbr>ermail/openstack-dev/2016-Nove<wbr>mber/107177.html</a><br>
</blockquote>
<br></div>
So...if step 1 is just enabling the placement service in all of the master branch jobs, I think that's probably easy enough.<br>
<br>
Here is an example of enabling services via devstack-gate:<br>
<br>
<a href="https://review.openstack.org/#/c/345626/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/345626/</a><br>
<br>
The problem would be if that's not controlled via branch. But I think it can be, so you can have the 'base' set of services enabled on all branches, and then you can filter by branch. We could probably do that just like how 'tlsproxy' is done here:<br>
<br>
<a href="https://github.com/openstack-infra/devstack-gate/blob/eb895ca90282019493c7889f57e8c4143468cfa9/features.yaml#L174" rel="noreferrer" target="_blank">https://github.com/openstack-i<wbr>nfra/devstack-gate/blob/eb895c<wbr>a90282019493c7889f57e8c4143468<wbr>cfa9/features.yaml#L174</a><br>
<br>
sdague would be the person to ask for sure.<br>
<br>
--<br>
<br>
The other thing we've talked about some this week is adding a 'ready-for-upgrade' nova-manage command which will do some basic sniff testing of your deployment and let you know if you're ready to start running online data migrations and schema migrations, etc. This would check some basic things like if placement is setup and we can make requests to the REST API from the command. Given the client making these requests today is in the compute nodes, the configuration is needed there, but I expect most people are running nova-manage from their control nodes, so I'm not sure how this is going to work, dansmith probably has ideas. My point is, it'd be odd for the command to require the placement config setup on the control node just for the nova-manage command to work, even though the control node doesn't use the placement REST API anywhere else (well except maybe the scheduler now in Ocata). Alternatively it's odd to run nova-manage from all of your compute nodes, but....maybe it makes sense to also be running that check on your compute nodes before upgrading them too. I'll stop rambling now.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
## Long Term Stuff<br>
</blockquote>
<br>
Honestly I skimmed this part because I'm mostly concerned with immediate priorities, but understand if you need to do a brain dump for posterity and tire kicking.<div class="elided-text"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
### Making Claims in the Placement API<br>
<br>
After Ocata the placement API will evolve to make claims, on the<br>
/allocations endpoint. When presented with a set of resources<br>
requirements _the_ resource provider that satisfies those requiements<br>
will be returned and the claim of resources made in a single step. To<br>
quote Mr. Pipes again:<br>
<br>
once we have a placement service actually doing claims, the<br>
returned resource providers for an allocation will be the actual<br>
resource providers that were allocated against (which include<br>
*both* compute node providers as well as any resource provider of<br>
a shared resource that was allocated)<br>
<br>
Just so folk are aware.<br>
<br>
### Moving Placement out of Nova<br>
<br>
If this is something we ever plan to do (there appear to be multiple<br>
points of view) then it is something we need to prepare for to ease<br>
the eventual transition. Some of these things include:<br>
<br>
* Removing as much 'nova.*' packages from the hierarchy of placement<br>
modules.<br>
* Getting the new placement DB[5][6] handled in some way<br>
* Removing remotable from the resource provider objects. The intent is<br>
that these will never be accessed other than through the HTTP API and<br>
since that scales horizontally, no RPC should be required. If we<br>
ever plan to remove it, sooner is better than later. A POC has been<br>
submitted[7], but there's disagreement about whether we should do<br>
it. We need to resolve that.<br>
<br>
[5] <a href="https://review.openstack.org/#/c/362766/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/362766/</a><br>
[6] <a href="https://etherpad.openstack.org/p/placement-optional-db-spec" rel="noreferrer" target="_blank">https://etherpad.openstack.org<wbr>/p/placement-optional-db-spec</a><br>
[7] <a href="https://review.openstack.org/#/c/404279/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/404279/</a><br>
<br>
# Pending Planned Work<br>
<br>
## Custom Resource Classes<br>
<br>
Jay just posted a big update[8] on that so go look at that. A lot of<br>
code has merged, but a lot of code[9] is still in flight.<br>
<br>
[8]<br>
<a href="http://lists.openstack.org/pipermail/openstack-dev/2016-December/108393.html" rel="noreferrer" target="_blank">http://lists.openstack.org/pip<wbr>ermail/openstack-dev/2016-Dece<wbr>mber/108393.html</a><br>
<br>
[9] <a href="https://review.openstack.org/#/q/topic:bp/custom-resource-classes" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/q/topic:bp/custom-resource-cl<wbr>asses</a><br>
<br>
## Filtering compute nodes with the placement API<br>
<br>
Already mentioned (with links) above.<br>
<br>
## Nested Resource Providers<br>
<br>
In discussions yesterday about Ocata priorities[10] we clarified that<br>
while resource providers matter, they are a stretch for Ocata. The<br>
primary goal is to have enough discussion and experimentation now so<br>
that we can have useful discussions at the PTG.<br>
<br>
The spec[11] has merged, the code is in a stack[12]. There's general<br>
agreement on the implementation, but there still seems to be some<br>
concern about how it is all going to work and what it all means in<br>
actual practice. The expectation is that we'll figure things when<br>
we're doing that actual practice.<br>
<br>
[10] <a href="https://review.openstack.org/#/c/404456/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/404456/</a><br>
[11]<br>
<a href="http://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/nested-resource-providers.html" rel="noreferrer" target="_blank">http://specs.openstack.org/ope<wbr>nstack/nova-specs/specs/ocata/<wbr>approved/nested-resource-<wbr>providers.html</a><br>
<br>
[12] <a href="https://review.openstack.org/#/c/377138/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/377138/</a><br>
<br>
## Allocations for generic PCI devices<br>
<br>
This code was abandoned because it was making some bad assumptions<br>
about how PCI device handling is done. See the abandoned review[13]<br>
for more information.<br>
<br>
[13] <a href="https://review.openstack.org/#/c/374681/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/374681/</a><br>
<br>
# Pending Pickup Work<br>
<br>
(Bugs[14], stuff from the leftovers etherpad[15], other random bits of<br>
improvement.)<br>
<br>
[14] <a href="https://bugs.launchpad.net/nova/+bugs?field.tag=placement" rel="noreferrer" target="_blank">https://bugs.launchpad.net/nov<wbr>a/+bugs?field.tag=placement</a><br>
[15] <a href="https://etherpad.openstack.org/p/placement-newton-leftovers" rel="noreferrer" target="_blank">https://etherpad.openstack.org<wbr>/p/placement-newton-leftovers</a><br>
<br>
* Demo inventory update script:<br>
<a href="https://review.openstack.org/#/c/382613/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/382613/</a><br>
<br>
This one might be considered a WIP because how it chooses to do<br>
things (rather simply and dumbly) may not be in line with expecations.<br>
<br>
* CORS support in placement API:<br>
<a href="https://review.openstack.org/#/c/392891/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/392891/</a><br>
<br>
John Garbutt's review led to finding a huge bug in this (service<br>
wouldn't start in an actual deployment). That's been fixed.<br>
<br>
* Handling limits in schema better<br>
<a href="https://review.openstack.org/#/c/399002/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/399002/</a> (needs review)<br>
<a href="https://review.openstack.org/#/c/398998/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/398998/</a> (needs fixes from submitter)<br>
<br>
* [WIP] Placement api: Add json_error_formatter to defaults<br>
<a href="https://review.openstack.org/#/c/395194/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/395194/</a><br>
<br>
This is an effort to avoid boilerplate, but no good solution has<br>
been determined yet. Reviewers can help us figure a good way to<br>
hande things.<br>
<br>
* Small improvements to placement.rst<br>
<a href="https://review.openstack.org/#/c/403811/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/403811/</a><br>
<br>
# End<br>
<br>
As usual, I hope this is useful to people. If something is missing<br>
or incorrect please say so. It's quite a bit of work to assemble this,<br>
but it's useful to me, so I'd be doing it anyway, even if I wasn't<br>
sending it out. I hope other people find it useful. If there's<br>
something I can do to make it more useful, let me know.<br>
</blockquote>
<br></div>
I think this is useful. Honestly I didn't read the last one, but I read this one, and it's something that helps me since there are a lot of moving parts going on with resource providers and I need to make sure we aren't starving parts of the priority work, especially as we're 2 weeks from o-2. Plus I generally don't make it to the weekly scheduler meeting so this is a nice recap of the weekly events.<font color="#888888"><br>
<br>
-- <br>
<br>
Thanks,<br>
<br>
Matt Riedemann</font><div class="elided-text"><br>
<br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
</div></blockquote></div><br></div></div></div>