Open Stack

On 12/17/2018 06:15 PM, Matt Riedemann wrote:
> On 12/12/2018 1:18 PM, Matt Riedemann wrote:
>> Coming back on this thread [1], I've got a partial fix up which I'm 
>> hoping will help:
>>
>> https://review.openstack.org/#/c/624778/
>>
>> That will avoid joining on some other tables depending on your 
>> configuration. It would be great if you could see if that helps 
>> resolve your issue. I think you just reverted 
>> https://review.openstack.org/#/c/276861/ as a workaround but it would 
>> be good to know if a more permanent fix (mine) gets you similar, or at 
>> least satisfactory, results.
>>
>> [1] 
>> http://lists.openstack.org/pipermail/openstack-dev/2018-October/thread.html#135941 
>>
> 
> I have abandoned that change since it turns out that we need to join on 
> the instance_system_metadata table to get the instance password which is 
> retrieved from a base metadata request. Otherwise you can see the 
> failures here [1].
> 
> So either we need to:
> 
> * Optimize the instance get DB query and joins we do. Dan was looking at 
> this but it was non-trivial.
> 
> * Reconsider how we store the instance password so it's not in the 
> instance_system_metadata table.
> 
> Or deployments can aggressively cache the metadata API responses (or 
> scale out the number of metadata API workers) to try and deal with load.
> 
> [1] 
> http://logs.openstack.org/78/624778/1/check/tempest-full/8d3c124/controller/logs/screen-n-api-meta.txt.gz#_Dec_13_08_45_56_602421 

Well, technically, we *could* do all three of the above, right? :) It's 
not an either/or situation, AFAIU.

 From looking at your patches, I think the long-term solution to this 
would be to stop storing the instance password in the 
instance_system_metadata table, but from looking into those code paths 
(eww...) I realize that would be a huge chunk of tech debt refactoring. 
Maybe something to line up for early Train?

Best,
-jay