[openstack-dev] [searchlight] OpenStack Newton Austin Summit
Tripp, Travis S
travis.tripp at hpe.com
Tue May 3 22:22:11 UTC 2016
Below is my summary of the Searchlight related discussions and results from the Austin Summit. I apologize for the length, but just decided to include all the session summaries in a single email. As always, please correct, add, etc!
5 Sessions (3 Searchlight Session, 1 joint Swift Team Session, 1 Horizon Session)
Searchlight Notifications (Fishbowl)
This had good attendance with at least 25 people in the room. We used the following etherpad to communicate and share:
Jay Pipes and another person (didn’t catch his name) provided a lot of help with pointers on using versioned notifications. Jay said that they (Nova) are working towards providing an API where we can get a schema  for the versioned notifications and then can extract the data we need based on the version of the Nova data that we need.
Jay later introduced us to Balazs Gibizer (gibi - the Nova versioned notification sub-team lead). Balazs said he’d love to help us get any data in that we need and he gave us the pointer that they are restarting the weekly versioned notification meetings .
It was mentioned that we also we should take a look at “Stackdistiller” .
Jay also mentioned that he has brought up the topic of the Nova team to stop trying to do their own search and proxy the Nova list API to Searchlight .
Ironic notifications: Three people from Ironic attended and expressed a lot of interest in supporting Searchlight. They noted that Ironic is currently working on adding notifications and their feedback was "Please look at the notifications patch and just tell us what you need.” 
Designate notifications: Graham Hayes noted that we need to consider v1 deprecated and can do that as soon as Horizon moves to v2.
Cinder notifications: Duncan Thomas says that the changes Searchlight needs for notifications should be available within a few weeks.
Heat notifications: Several people expressed interest in Heat, but there weren’t any concrete actions taken from this session.
I communicated that the most important theme for Newton is production readiness. We will be targeting moving from 0.x to 1.x this release (Kilo was an experimental glance feature, Liberty was Searchlight 0.1, and Mitaka was Searchlight 0.2). So scalability, security, and performance are all top priority. This includes moving to ElasticSearch 2.x, which is nearly complete. Our ongoing theme of adding plugins for additional resources will continue, but reviews related to production readiness should have higher priority. Richard Jones mentioned using OSIC for testing and would talk to somebody about creating OSAD scripts to deploy Searchlight in OSIC.
The session was unfortunately a bit short, leaving us with only time to walk the list of all blueprints listed as High and give people an opportunity to voice an opinion whether or not any high priorities should be bumped down and whether or not we were missing any high priorities. Here are a few notes from that session:
We dropped the priority of adding support for Neutron policy based sharing to medium . Brad Pokorny (Symantec – had a main conference presentation on securing APIs via policy) told us that he thought this should be a lower priority than our other blueprints. He also said he’d be willing to look over our current Policy controls on searchlight to look for holes.
We added back a story to further improve developer docs. Several people asked how they could create a plugin and said they wanted more help. Steve has already started on this request .
We agreed on the following components:
* Provide a middleware pipeline plugin with direct injection using ES library
* Provide a hook in the container sync work to also interact with the ES injection library
Swift Middleware pipeline plugin
We agreed with the team to continue building out the ES indexing library and it is okay to have a dependency on the ES client library. In discussions about OSLO messaging, we explored the idea to build ES as a driver for the messaging (instead of rabbit). This would allow deployers with Rabbit to use Rabbit and those who don’t to go straight to ES. However, there was concern that the dependencies  that OSLO messaging pulls in are too much for the Swift-only deployers like Swiftstack to handle. Swiftstack in particular spoke of how they would have to package and support each of those libraries for 7 different OS distributions. They did say that if taking “rabbit” out as a default driver for oslo.messaging would remove most the dependencies, then this could be reconsidered.
Action: See how much the oslo.messaging dependencies are reduced when the rabbit driver isn’t included.
There was discussion about increasing performance of indexing by only indexing in bulk. The idea was suggested to use Green threads to implement a very simple batch queuing system. It would just collect incoming data requests and given a certain configurable number (e.g. 20, 200, etc) at which point it would flush the queue and send a batch of requests to ES. It was agreed this could be a follow-on patch.
The spec  for this library will be updated appropriately.
Swift Container Sync
This is something which Timur Alperovich is working on and was originally just going to do internal swift-y things . However, they’ve decided that they would make it a bit smarter and able to handle plugging the direct ES indexer library into it. It would then provide backup indexing to the middleware (cache coherency kind of concepts).
Searchlight Pipeline Architecture
Yuntong Jin from Intel was able to attend the session and he did a quick whiteboard diagram on the concept. We agreed in principle to the concepts and we agreed that we need to start merging by Newton 2 to avoid destabilization at the end of Newton.
Searchlight Cross Region Search
General agreement was that this was a deployer choice. We should put region information into the data we index and document how deployers could set up Tribe nodes. However, there are other complications and we decided to lower the priority of doing any more work this release. We may do more in the future. The spec has been updated to reflect this decision .
Searchlight Data Normalization
We presented the problem with ElasticSearch 2.0  and discussed various solutions, but didn’t reach a perfect answer. Solutions discussed:
* Different indexes when there are collisions
* Same index and normalize for storage, but massage data back into API format on the way out (current choice and patch up)
* Use multi-fields (Jason Galyon from Softlayer said he would email us with additional thoughts here)
Searchlight in Horizon
Registry and Actions
The resource type registry related work was given “Critical” Priority. The developer panel and various actions were then spread across different priorities . This is important for the searchlight-ui extensibility.
Items of particular note for Searchlight where we can help are that several people complained about the lack of pagination and search in Horizon .
[14o] Full notes here: https://etherpad.openstack.org/p/horizon-newton-feedback
More information about the OpenStack-dev