Open Stack

Hi Adrian,

Yes, you've covered some of what my soon-to-send "API implementation"
email already covers. :)

On Mon, Feb 14, 2011 at 06:32:40PM +0000, Adrian Otto wrote:
> Ordered delivery of messages is a feature that I recognize is central to the design of a queue service. A queue must be durable (persistent) in order to provide reliable ordering capability. Having this feature simplifies implementation of applications that consume the queue. I agree this feature should be optional, but available regardless of the protocol used to access the service.

Agreed, and this is something I've already given a lot of though and
have options drafted out for in the API.

> I also feel strongly that there should be a concept of a sensible trade-off between performance and reliability/durability. For example, allowing modes of operation ranging from best-effort (fast) to guaranteed ordered delivery with optional error/timeout callback (slow/reliable). For those requiring top speed, accommodations for unreliable (in-memory, un-ordered) queues can be made within the application.

Also agree, and these will be message parameters. Initially this will
just be a 'persistent' boolean to keep it simple, but we can expand
on this in the future.

> I also feel that the (de)queueing logic should be modular, and selectable at the time an item is de-queued. This can allow advanced queue theory implementations without haciking. Specifically, allow fine control over what items are in what order, and under what conditions. One example of this would be a simple implementation of FIFO by default, and an optional priority-queue implementation as a selectable alternative. Having a well defined plug-in interface for this should prove to be very useful when custom algorithms are desired.

I'd like to hear more once I send the API draft, but I'm currently
thinking of allowing workers (which is really just a client issuing
GETs instead of PUTs), be able to specify if the message should be
hidden on GET, and for how long. When the worker is done, it can
issue a DELETE (job is complete) or modify the message to rest hidden
status, timeout, and so forth. Essentially 'reinsert with delay'
semantics. I'll try to get the API draft out soon (like today) so we
can continue this discussion with more context.

> Although now I'm diving into what really may be considered the implementation discussion, I think that any queued item should be allowed to have arbitrary metadata attached to it that's automatically indexed for internal search capability. Think headers, not body.

I think this may be a great v2 feature, but initially I want to focus
only on the queue name. Other metadata and secondary indexes start
to add complexity, which I agree will be needed, but may want to hold
off for v1.

Along with the API email, I also have an initial roadmap email of
features so we can add and prioritize various queue features (like
this) with various milestones. I want to start very simple and work
incrementally, but also want to make sure we fully understand these
and other advanced ideas so we can plan appropriately.

-Eric

> 
> Adrian
> 
> On Feb 14, 2011, at 9:51 AM, Eric Day wrote:
> 
> > Hi everyone,
> > 
> > When looking at other services to include as part of OpenStack, the
> > first that comes to mind for many is a queue. A queue service can
> > not only provide a useful public cloud service, but can also provide
> > one of the building blocks for other services. I've been leading an
> > effort to research and gather requirements for a queue service and
> > I'd like to share the current state and get community feedback. I
> > expect real development to begin very soon, and would also like to
> > identify developers who will have time to dedicate to this project.
> > 
> > I'd like to note this is not an official OpenStack project yet. The
> > intention is once we have the community support and a simple
> > implementation, we will submit the project to the OpenStack Project
> > Oversight Committee for approval.
> > 
> > The reason we are initiating our own project vs using an existing one
> > is due to simplicity, modularity, and scale. Also, very few (if any)
> > existing queue systems out there were built with multi-tenant cloud
> > use cases in mind. Very few also have a simple and extensible REST
> > API. There are possible solutions to build an AMQP based service,
> > but AMQP brings complexity and a protocol not optimized for high
> > latency and intermittent connectivity.
> > 
> > The primary goals of the queue service are:
> > 
> > * Simple - Think simple REST based queues for most use cases. Easy
> >  to access and use from any language. It should not require much
> >  setup, if any, before you can start pushing messages into it.
> > 
> > * Modular API - Initially we'll focus on a simple REST API,
> >  but this will not in any way be a first-class API. It should be
> >  possible to add other protocols (AMQP, protocol buffers, Gearman,
> >  etc) for other use cases. Note that the internal service API will
> >  not always provide a 1-1 mapping with the external API, so some
> >  features with advanced protocols may be unavailable.
> > 
> > * Fast - Since this will act as a building block for other services
> >  that my drive heavy throughput, performance will have a focus. This
> >  mostly comes down to implementation language and how clients and
> >  workers interact with the broker to reduce network chatter.
> > 
> > * Multi-tenant - Support multiple accounts for the service, and since
> >  this will also be a public service for some deployments, protect
> >  against potentially malicious users.
> > 
> > * Persistent - Allow messages to optionally be persistent. For
> >  protocols that can support it, this can be an optional flag while
> >  the message is submitted. The persistent storage should also be
> >  modular so we can test various data stores and accommodate different
> >  deployment options.
> > 
> > * Zones and locality awareness - As we've been discussing in other
> >  threads, locality in cloud services is an important feature. When
> >  dealing with where messages should be processed, we need to have
> >  location awareness to process data where it exists to reduce network
> >  overhead and processing time.
> > 
> > Before diving down into implementation details, I would like to hear
> > what folks have to say about the initial requirements above. Once
> > there is something along the lines of agreement, I'll be sending out
> > other topics for discussion dealing with implementation.
> > 
> > I'm looking forward to your feedback. Thanks!
> > 
> > -Eric
> > 
> > _______________________________________________
> > Mailing list: https://launchpad.net/~openstack
> > Post to     : openstack at lists.launchpad.net
> > Unsubscribe : https://launchpad.net/~openstack
> > More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
> Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of the
> individual or entity to which this message is addressed, and unless otherwise
> expressly indicated, is confidential and privileged information of Rackspace.
> Any dissemination, distribution or copying of the enclosed material is prohibited.
> If you receive this transmission in error, please notify us immediately by e-mail
> at abuse at rackspace.com, and delete the original message.
> Your cooperation is appreciated.
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp