Open Stack

Responses to some of Jay's comments below, but first, to keep this
on track with the original goal of the thread ("How to debug no
valid host failures with placement") before I drag it to the side,
some questions.

When people ask for something like what Chris mentioned:

     hosts with enough CPU: <list1>
     hosts that also have enough disk: <list2>
     hosts that also have enough memory: <list3>
     hosts that also meet extra spec host aggregate keys: <list 4>
     hosts that also meet image properties host aggregate keys: <list 5>
     hosts that also have requested PCI devices: <list 6>

What are the operational questions that people are trying to answer
with those results? Is the idea to be able to have some insight into
the resource usage and reporting on and from the various hosts and
discover that things are being used differently than thought? Is
placement a resource monitoring tool, or is it more simple and
focused than that? Or is it that we might have flavors or other
resource requesting constraints that have bad logic and we want to
see at what stage the failure is?  I don't know and I haven't really
seen it stated explicitly here, and knowing it would help.

Do people want info like this for requests as they happen, or to be
able to go back later and try the same request again with some flag
on that says: "diagnose what happened"?

Or to put it another way: Before we design something that provides
the information above, which is a solution to an undescribed
problem, can we describe the problem more completely first to make
sure that what solution we get is the right one. The thing above,
that set of information, is context free.

On Wed, 1 Aug 2018, Jay Pipes wrote:
> On 08/01/2018 02:02 PM, Chris Friesen wrote:
>> I think the only way to get useful info on a failure would be to break down 
>> the huge SQL statement into subclauses and store the results of the 
>> intermediate queries.
>
> This is a good idea and something that can be done.

I can see how it would be a good idea from an explicit debugging
standpoint, but is it a good idea on all fronts? From the very early
days when placement was just a thing under your pen on whiteboards,
we were trying to achieve something that wasn't the FilterScheduler
but achieved efficiencies and some measure of black boxed-ness by
being as near as possible to a single giant SQL statement as we
could get it. Do we want to get too far away from that?

Another thing to consider is that in a large installation, logging
these intermediate results (if done in the listing-hosts way
indicated above) would be very large without some truncating or
"only if < N results" guards.

Would another approach be to make it easy to replay a resource
request that incrementally retries the request with a less
constrained set of requirements (expanding by some heuristic we
design)? Something on a different URI where the response is in
neither of what /allocation_candidates or /resourcer_providers
returns, but allows the caller to know the boundary of results and
no results is.

One could also imagine a non-http interface to placement that
outputs something a bit like 'top': a regularly updating scan of
resource usage. But it's hard to know if that is even relevant
without more info as asked above.

It could very well be that explicit debugging of filtering stages is
the right way to go, but we should look closely at the costs of
doing so. Part of me is all: Please, yes, let's do it, it would make
the code _so_ much more comprehensible. But there were reasons we
made the complex SQL in the first place.

> Unfortunately, it's refactoring work and as a community, we tend to 
> prioritize fancy features like NUMA topology and CPU pinning over refactoring 
> work.

I think if we, as a community, said "no", that would be okay. That's
really all it would take. We effectively say "no" to features all the
time anyway, because we've generated software to which it takes 3
years to add something like placement to anyway, for very little
appreciable gain in that time (Yes there are many improvements under
the surface and with things like race conditions, but in terms of
what can be accomplished with the new tooling, we're still not
there).

If our labour is indeed valuable we can choose to exercise greater
control over its direction.

-- 
Chris Dent                       ٩◔̯◔۶           https://anticdent.org/
freenode: cdent                                         tw: @anticdent