On 12/12/2018 1:18 PM, Matt Riedemann wrote: > Coming back on this thread [1], I've got a partial fix up which I'm > hoping will help: > > https://review.openstack.org/#/c/624778/ > > That will avoid joining on some other tables depending on your > configuration. It would be great if you could see if that helps resolve > your issue. I think you just reverted > https://review.openstack.org/#/c/276861/ as a workaround but it would be > good to know if a more permanent fix (mine) gets you similar, or at > least satisfactory, results. > > [1] > http://lists.openstack.org/pipermail/openstack-dev/2018-October/thread.html#135941 I have abandoned that change since it turns out that we need to join on the instance_system_metadata table to get the instance password which is retrieved from a base metadata request. Otherwise you can see the failures here [1]. So either we need to: * Optimize the instance get DB query and joins we do. Dan was looking at this but it was non-trivial. * Reconsider how we store the instance password so it's not in the instance_system_metadata table. Or deployments can aggressively cache the metadata API responses (or scale out the number of metadata API workers) to try and deal with load. [1] http://logs.openstack.org/78/624778/1/check/tempest-full/8d3c124/controller/logs/screen-n-api-meta.txt.gz#_Dec_13_08_45_56_602421 -- Thanks, Matt