Open Stack

Hi everyone,

After previous summits where we had vertical tracks for Nova sessions I 
would provide a recap for each session.

The Forum in Boston was a bit different, so here I'm only attempting to 
recap the Forum sessions that I ran. Dan Smith led a session on Cells 
v2, John Garbutt led several sessions on the VM and Baremetal platform 
concept, and Sean Dague led sessions on hierarchical quotas and API 
microversions, and I'm going to leave recaps for those sessions to them.

I'll do these one at a time in separate emails.


Using Searchlight to list instances across cells in nova-api
------------------------------------------------------------

The etherpad for this session is here [1]. The goal for this session was 
to explain the problem and proposed plan from the spec [2] to the 
operators in the room and get feedback.

Polling the room we found that not many people are deploying Searchlight 
but most everyone was using ElasticSearch.

An immediate concern that came up was the complexity involved with 
integrating Searchlight, especially around issues with latency for state 
changes and questioning how this does not redo the top-level cells v1 
sync issue. It admittedly does to an extent, but we don't have all of 
the weird side code paths with cells v1 and it should be self-healing. 
Kris Lindgren noted that the instance.usage.exists periodic notification 
from the computes hammers their notification bus; we suggested he report 
a bug so we can fix that.

It was also noted that if data is corrupted in ElasticSearch or is out 
of sync, you could re-sync that from nova to searchlight, however, 
searchlight syncs up with nova via the compute REST API, which if the 
compute REST API is using searchlight in the backend, you end up getting 
into an infinite loop of broken. This could probably be fixed with 
bypass query options in the compute API, but it's not a fun problem.

It was also suggested that we store a minimal set of data about 
instances in the top-level nova API database's instance_mappings table, 
where all we have today is the uuid. Anything that is set in the API 
would probably be OK for this, but operators in the room noted that they 
frequently need to filter instances by an IP, which is set in the 
compute. So this option turns into a slippery slope, and is potentially 
not inter-operable across clouds.

Matt Booth is also skeptical that we can't have a multi-cell query 
perform well, and he's proposed a POC here [3]. If that works out, then 
it defeats the main purpose for using Searchlight for listing instances 
in the compute API.

Since sorting instances across cells is the main issue, it was also 
suggested that we allow a config option to disable sorting in the API. 
It was stated this would be without a microversion, and filtering/paging 
would still be supported. I'm personally skeptical about how this could 
be consider inter-operable or discoverable for API users, and would need 
more thought and input from users like Monty Taylor and Clark Boylan.

Next steps are going to be fleshing out Matt Booth's POC for efficiently 
listing instances across cells. I think we can still continue working on 
the versioned notifications changes we're making for searchlight as 
those are useful on their own. And we should still work on enabling 
searchlight in the nova-next CI job so we can get an idea for how the 
versioned notifications are working by a consumer. However, any major 
development for actually integrating searchlight into Nova is probably 
on hold at the moment until we know how Matt's POC works.

[1] 
https://etherpad.openstack.org/p/BOS-forum-using-searchlight-to-list-instances
[2] 
https://specs.openstack.org/openstack/nova-specs/specs/pike/approved/list-instances-using-searchlight.html
[3] https://review.openstack.org/#/c/463618/

-- 

Thanks,

Matt