Open Stack

Hello everybody,

Below is my summary of the Searchlight related discussions and results from the Austin Summit. I apologize for the length, but just decided to include all the session summaries in a single email. As always, please correct, add, etc!

5 Sessions (3 Searchlight Session, 1 joint Swift Team Session, 1 Horizon Session)

https://www.openstack.org/summit/austin-2016/summit-schedule/global-search?t=SearchLight%3A



Searchlight Notifications (Fishbowl)

This had good attendance with at least 25 people in the room. We used the following etherpad to communicate and share:

https://etherpad.openstack.org/p/searchlight-newton-summit-notifications

Nova notifications:

Jay Pipes and another person (didn’t catch his name) provided a lot of help with pointers on using versioned notifications. Jay said that they (Nova) are working towards providing an API where we can get a schema [0] for the versioned notifications and then can extract the data we need based on the version of the Nova data that we need.

[0] http://eavesdrop.openstack.org/#Nova_Notification_Meeting

Jay later introduced us to Balazs Gibizer (gibi - the Nova versioned notification sub-team lead). Balazs said he’d love to help us get any data in that we need and he gave us the pointer that they are restarting the weekly versioned notification meetings [1].
  

[1] https://review.openstack.org/#/c/311194/

It was mentioned that we also we should take a look at “Stackdistiller” [2].

[2] https://github.com/openstack/stacktach-stackdistiller

Jay also mentioned that he has brought up the topic of the Nova team to stop trying to do their own search and proxy the Nova list API to Searchlight [3].

[3] http://lists.openstack.org/pipermail/openstack-dev/2016-April/093482.html


Ironic notifications: Three people from Ironic attended and expressed a lot of interest in supporting Searchlight. They noted that Ironic is currently working on adding notifications and their feedback was "Please look at the notifications patch and just tell us what you need.” [4]

[4] https://bugs.launchpad.net/searchlight/+bug/1526408


Designate notifications: Graham Hayes noted that we need to consider v1 deprecated and can do that as soon as Horizon moves to v2.

Cinder notifications: Duncan Thomas says that the changes Searchlight needs for notifications should be available within a few weeks.

Heat notifications: Several people expressed interest in Heat, but there weren’t any concrete actions taken from this session.


Searchlight Priorities

I communicated that the most important theme for Newton is production readiness. We will be targeting moving from 0.x to 1.x this release (Kilo was an experimental glance feature, Liberty was Searchlight 0.1, and Mitaka was Searchlight 0.2). So scalability, security, and performance are all top priority. This includes moving to ElasticSearch 2.x, which is nearly complete. Our ongoing theme of adding plugins for additional resources will continue, but reviews related to production readiness should have higher priority. Richard Jones mentioned using OSIC for testing and would talk to somebody about creating OSAD scripts to deploy Searchlight in OSIC.

The session was unfortunately a bit short, leaving us with only time to walk the list of all blueprints listed as High and give people an opportunity to voice an opinion whether or not any high priorities should be bumped down and whether or not we were missing any high priorities. Here are a few notes from that session:

Melissa, the Rackspace Public Cloud control panel program manager attended and said that they are looking into using Searchlight to fill the API gaps. They have tried to handle quite a few things on the front end (javascript), but still have some troubles getting what they need out of the APIs. She said her highest priority for Searchlight are servers (instances) and doing anything we can to ensure there is no impact to Nova when deploying Searchlight. We mentioned the versioned notification work and I also proposed that we add a configuration option to disable API callbacks for any data not received via notifications.  I have opened a bug on this, and we’ll have to look further into this idea [5].

[5] https://bugs.launchpad.net/searchlight/+bug/1577947

We dropped the priority of adding support for Neutron policy based sharing to medium [6]. Brad Pokorny (Symantec – had a main conference presentation on securing APIs via policy) told us that he thought this should be a lower priority than our other blueprints. He also said he’d be willing to look over our current Policy controls on searchlight to look for holes.

[6] https://blueprints.launchpad.net/searchlight/+spec/neutron-tenant-rbac

We added back a story to further improve developer docs.  Several people asked how they could create a plugin and said they wanted more help. Steve has already started on this request [7].

[7] https://review.openstack.org/#/c/312224/



Swift Integration

We agreed on the following components:

* Provide a middleware pipeline plugin with direct injection using ES library
* Provide a hook in the container sync work to also interact with the ES injection library

Swift Middleware pipeline plugin

We agreed with the team to continue building out the ES indexing library and it is okay to have a dependency on the ES client library.  In discussions about OSLO messaging, we explored the idea to build ES as a driver for the messaging (instead of rabbit). This would allow deployers with Rabbit to use Rabbit and those who don’t to go straight to ES. However, there was concern that the dependencies [8] that OSLO messaging pulls in are too much for the Swift-only deployers like Swiftstack to handle. Swiftstack in particular spoke of how they would have to package and support each of those libraries for 7 different OS distributions. They did say that if taking “rabbit” out as a default driver for oslo.messaging would remove most the dependencies, then this could be reconsidered.

Action: See how much the oslo.messaging dependencies are reduced when the rabbit driver isn’t included. 

[8] https://pypi.python.org/pypi/oslo.messaging


There was discussion about increasing performance of indexing by only indexing in bulk. The idea was suggested to use Green threads to implement a very simple batch queuing system.  It would just collect incoming data requests and given a certain configurable number (e.g. 20, 200, etc) at which point it would flush the queue and send a batch of requests to ES.  It was agreed this could be a follow-on patch.

The spec [9] for this library will be updated appropriately.

[9] https://review.openstack.org/#/c/305404/

Swift Container Sync

This is something  which Timur Alperovich is working on and was originally just going to do internal swift-y things [10].  However, they’ve decided that they would make it a bit smarter and able to handle plugging the direct ES indexer library into it. It would then provide backup indexing to the middleware (cache coherency kind of concepts).

[10] http://docs.openstack.org/developer/swift/overview_container_sync.html#what-s-going-on-behind-the-scenes-in-the-cluster


Searchlight Pipeline Architecture

Yuntong Jin from Intel was able to attend the session and he did a quick whiteboard diagram on the concept. We agreed in principle to the concepts and we agreed that we need to start merging by Newton 2 to avoid destabilization at the end of Newton.


Searchlight Cross Region Search

General agreement was that this was a deployer choice.  We should put region information into the data we index and document how deployers could set up Tribe nodes.  However, there are other complications and we decided to lower the priority of doing any more work this release. We may do more in the future. The spec has been updated to reflect this decision [11].

[11] https://review.openstack.org/#/c/301227/


Searchlight Data Normalization

We presented the problem with ElasticSearch 2.0 [12] and discussed various solutions, but didn’t reach a perfect answer. Solutions discussed:

* Different indexes when there are collisions
* Same index and normalize for storage, but massage data back into API format on the way out (current choice and patch up)
* Use multi-fields (Jason Galyon from Softlayer said he would email us with additional thoughts here)

[12] https://bugs.launchpad.net/searchlight/+bug/1532010


Searchlight in Horizon

Registry and Actions

The resource type registry related work was given “Critical” Priority. The developer panel and various actions were then spread across different priorities [13]. This is important for the searchlight-ui extensibility.

[13] https://etherpad.openstack.org/p/horizon-newton-priorities

Operator Feedback

Items of particular note for Searchlight where we can help are that several people complained about the lack of pagination and search in Horizon [14].

[14o] Full notes here: https://etherpad.openstack.org/p/horizon-newton-feedback