Open Stack

Hello Swift team,

I sent out a summary of what I believe was discussed at our joint session to just [searchlight] and wanted to copy it to the [swift] thread so that we are on the same page.  Please take a look and reply with any comments, corrections, questions, etc.  Thanks!

Swift Integration

We agreed on the following components:

* Provide a middleware pipeline plugin with direct injection using ES library
* Provide a hook in the container sync work to also interact with the ES injection library

Swift Middleware pipeline plugin

We agreed with the team to continue building out the ES indexing library and it is okay to have a dependency on the ES client library.  In discussions about OSLO messaging, we explored the idea to build ES as a driver for the messaging (instead of rabbit). This would allow deployers with Rabbit to use Rabbit and those who don’t to go straight to ES. However, there was concern that the dependencies [8] that OSLO messaging pulls in are too much for the Swift-only deployers like Swiftstack to handle. Swiftstack in particular spoke of how they would have to package and support each of those libraries for 7 different OS distributions. They did say that if taking “rabbit” out as a default driver for oslo.messaging would remove most the dependencies, then this could be reconsidered.

Action: See how much the oslo.messaging dependencies are reduced when the rabbit driver isn’t included.

[8] https://pypi.python.org/pypi/oslo.messaging

There was discussion about increasing performance of indexing by only indexing in bulk. The idea was suggested to use Green threads to implement a very simple batch queuing system.  It would just collect incoming data requests and given a certain configurable number (e.g. 20, 200, etc) at which point it would flush the queue and send a batch of requests to ES.  It was agreed this could be a follow-on patch.

The spec [9] for this library will be updated appropriately.

[9] https://review.openstack.org/#/c/305404/

Swift Container Sync

This is something  which Timur Alperovich is working on and was originally just going to do internal swift-y things [10].  However, they’ve decided that they would make it a bit smarter and able to handle plugging the direct ES indexer library into it. It would then provide backup indexing to the middleware (cache coherency kind of concepts).

[10] http://docs.openstack.org/developer/swift/overview_container_sync.html#what-s-going-on-behind-the-scenes-in-the-cluster