Thanks Matt for the excellent suggestions in this email and the prior one.
I am currently trying to eliminate them one by one and will update.

Yes, by  forced host I do mean creating the server with an availability zone in the 
ZONE:NODE format.   Yes, I understand the scheduler filters aren't run but why should that be
an issue? 

For now, I am tracing all the logs from the PaaS layer all the way to Openstack nova placement API to
see if there is anything unusual.

Thanks,
Fred.

On Thursday, November 14, 2019, 10:07:15 AM PST, Matt Riedemann <mriedemos@gmail.com> wrote:


On 11/14/2019 10:09 AM, fsbiz@yahoo.com wrote:
> The requests coming in are "forced host" requests.  The PaaS layer
> maintains
> an inventory of actual bare-metal available nodes and a user has to
> explicitly select
> a baremetal node.  The PaaS layer then makes a nova api call for an
> instance to be created
> on that specific baremetal node.

To be clear, by forced host you mean creating the server with an
availability zone in the format ZONE:HOST:NODE or ZONE:NODE where NODE
is the ironic node UUID, correct?

https://docs.openstack.org/nova/latest/admin/availability-zones.html#using-availability-zones-to-select-hosts

Yeah that's a problem because then the scheduler filters aren't run.

A potential alternative is to create the server using a
hypervisor_hostname query hint that will run through the JsonFilter:

https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#jsonfilter

Then at least you're not forcing the node and run the scheduler filters.

I forget exactly how the scheduler code works in Queens with respect to
forced hosts/nodes on server create but the scheduler still has to
allocate resources in placement. It looks like we work around that in
Queens by disabling the limit we place on getting allocation candidates
from placement:

https://review.opendev.org/#/c/584616/

My guess is your PaaS layer has bugs in it since it's allowing users to
select hosts that are already consumed, or it's just racy. Anyway, this
is why nova uses placement since Pike for atomic consumption of
resources during scheduling.


--

Thanks,


Matt