Open Stack

Greetings,

The Zaqar team has been quiet in the last cycle but that doesn't mean
it's gone. This email is to share with you all what we've been focused
on during Kilo.

The team has been working on 3 main areas (order does not reflect
priority/importance):

* Relaxing some of the storage constraints by relying more on storage
capabilities and less on the guarantees. [0]

* Implement notifications to fulfill that missing part of the
service's goals. [1]

* Work on a non-wsgi transport[2]

Each of the above areas required important changes in the service that
I'll try to sumarize now:

Relaxing Storage Constraints
============================

As some of you know, the service has support for something we call
"flavors". These flavors difer from what we know as flavors in
services like Nova. The flavors in Zaqar, represent the type of
storage a set of messages will be stored into.

Every flavor in Zaqar has a set of capabilities. These capabilities
represent what the storage can/cannot do. For example, durability is
considered a capability and so are high-throughput and claims.

As of Juno, these capabilities were manually created but this has
changed. The capabilities are now introspected from the pools asigned
to the flavors and they are now a read-only field.

In addition, as part of this work, FIFO has been made optional and it
can now be enabled through the pool. [3]

To know a bit more about flavors and pools, please read this post. [4]

Implementing notifications
==========================

Fei Long Wang (flwang) has done an amazing job on getting this done. The
feature itself was quite big and it required patches in several
places. If you're interested in the details of the feature, please,
read the spec[1].

If you're going to use notifications, please, bare in mind that the
feature itself is quite new and it requires some extra testing
(especiall scale tests ;).

Non persistent transport
========================

Victoria Martinez de la Cruz (vkmc) has dedicated efforts to this
feature. She's done an amazing research on this field to find the best
way to make this work with Zaqar. During the cycle several websocket
libraries were examined and we ended up choosing atobhan. Likewise,
the protocol went through some iterations until we finally reached an
agreement.

This feature, however, didn't make it entirely in Kilo. Part of it
will be completed during Liberty. To be precise, it's possible to
manage queue's and connect to Zaqar using websockets but it's still
not possible to fully operate the API.

We felt that it was not right to rush this in at its current state and
that it'd be a good thing to have part of the feature as a preview for
some folks to play with it.

Move QueueController to the control plane
=========================================

This might be confusing for some folks that are not familiar with
Zaqar's architecture but I think it's also worth mentioning.

TL;DR: Zaqar has 2 data planes. 1 used for the actual data (messages)
and one used for the "control" - metadata- data (subscriptions information,
flavors, etc). These 2 planes can either use the same DB or a
different one.

Queue's used to live in the data plane but we've moved them to the
control plane. The main reason is that queue's are a lazy resource in
Zaqar and you don't really need to create them manually. The only
reason you'd create a queue manually is to set some metadata for it
(assign a flavor to a queue, for example). Since this is considered to
be metadata, we got to the conclusion there was no value on keeping
queue's in the data plane. This move brings other benefits like saving
space on the data store, avoid hitting the data store for queue's
lookups and management, pick better stores for each plane, etc.

I'd like to thank Shaifali Agrawal for her persistence and hard work
on this task. It was not an easy task, it required internal
refactoring, configuration changes and lots of fights with gerrit.
Thanks again!

The above changes, since they required breaking backwards compatibily,
have been implemented in a new version of the API, v2. The v2 follows
closely what existed already in v1.1 but it also has support for these
new features and the breaking changes I've mentioned already.

Unfortunately, as many of you know, our team has shrunk and each of us
work on other projects too. This means that there's little to none
client support for the above mentioned features. However, we'll be
working on bringing the client up-to-speed.

The team spent the entire cycle heads-down silently working on these
tasks in the best way possible. For the next cycles, I believe the
main focus will be in improving the adoption of the service and
improving UX besides fixing bugs, obviously.

Hope you find this summary useful and please, don't hesitate to shoot
questiongs if you've any.

Flavio

[0] https://github.com/openstack/zaqar-specs/blob/master/specs/kilo/approved/storage-capabilities.rst
[1] https://github.com/openstack/zaqar-specs/blob/master/specs/kilo/approved/notification-api.rst
[2] https://github.com/openstack/zaqar-specs/blob/master/specs/kilo/approved/persistent-transport.rst
[3] https://github.com/openstack/zaqar-specs/blob/master/specs/kilo/approved/fifo-optional.rst
[4] http://blog.flaper87.com/post/zaqar-pools-explained/
[5] https://blueprints.launchpad.net/zaqar/+spec/split-data-and-control-plane

-- 
@flaper87
Flavio Percoco
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150402/cff7597f/attachment.pgp>