[Openstack] Queue Service

Adrian Otto adrian.otto at rackspace.com
Tue Feb 15 17:47:23 UTC 2011


I think we should include the account string regardless if the auth headers are present, just like we do in the swift API.

You can still cache entities without the account string in the url, but it means that you need to add the account string (or some unique derivative) from the auth headers into your hash (stores and lookups in your cache must use the same scheme). That means that you must use a cache that has this capability. When you are using a simple cache that simply uses a hash of the URL, then Glen's concern below is absolutely valid.

It's best not to have multiple different URL's for the same entity, because when you do have a system architecture that relies on caching, you typically end up with an inefficient cache because the same entity is cached under multiple different URL's. If the URL is always the same for an identical entity, then you can eliminate that waste.

Furthermore, it's sensible to keep it simple. I'd rather eat the performance penalty of transmitting and parsing longer HTTP requests in order to reap the benefit of having less confused developers and system admins/operators. If I were tasked with debugging a malfunctioning queue system, I would welcome account strings in the URL's to help simplify my detective work. Simple access log files are much more useful that way.

Adrian

On Feb 15, 2011, at 7:45 AM, Glen Campbell wrote:

> Having the <account> in the URL makes the resource cacheable (via HTTP, I
> mean; you can always manually cache it, but that's not RESTful); without
> it, you would need to require unique <container>s to uniquely reference a
> cached copy.
> 
> On 2/14/11 7:57 PM, "Paul Voccio" <paul.voccio at rackspace.com> wrote:
> 
>> Looking at the swift docs, they reference a container like so:
>> 
>> METHOD /v1/<account>/<container> HTTP/1.1
>> 
>> http://docs.rackspacecloud.com/files/api/v1/cf-devguide-20110112.pdf
>> 
>> For the openstack api, it also includes the account id in the request:
>> 
>> POST /v1.1/214412/images HTTP/1.1
>> Host: servers.api.openstack.org
>> Content-Type: application/json
>> Accept: application/xml
>> X-Auth-Token: eaaafd18-0fed-4b3a-81b4-663c99ec1cb
>> 
>> 
>> 
>> This seems a bit different than what you're suggesting. What am I missing?
>> Shouldn't the account id be in the request with the auth headers to stay
>> in line with the other specs?
>> 
>> 
>> On 2/14/11 6:56 PM, "Eric Day" <eday at oddments.org> wrote:
>> 
>>> On Tue, Feb 15, 2011 at 12:49:01AM +0000, Paul Voccio wrote:
>>>> Dropping the account_id, would this also assume that there can be more
>>>> than one queue per account?
>>> 
>>> Yeah. Think of the queue name as an extra namespace layer much like
>>> a swift container, except you don't create or delete them, they just
>>> exist when there is an active message in it.
>>> 
>>> -Eric
>>> 
>>>> On 2/14/11 5:54 PM, "Eric Day" <eday at oddments.org> wrote:
>>>> 
>>>>> Yeah, for anonymous access that would be the case. Those are not
>>>> needed
>>>>> when the request comes in with OpenStack Auth headers (like nova).
>>>>> 
>>>>> Do you think we should be repeating the account id in the URI when
>>>>> the auth headers are present?
>>>>> 
>>>>> -Eric
>>>>> 
>>>>> On Mon, Feb 14, 2011 at 11:44:58PM +0000, Paul Voccio wrote:
>>>>>> Eric,
>>>>>> 
>>>>>> Just looking at the http operations. Shouldn't the calls be around
>>>> the
>>>>>> account then the queue?
>>>>>> 
>>>>>> GET /$account_id/queue/id to list all the queues
>>>>>> GET /$account_id/queue/id/message/id
>>>>>> 
>>>>>> Am I off here? Thoughts?
>>>>>> 
>>>>>> pvo
>>>>>> 
>>>>>> 
>>>>>> On 2/14/11 5:07 PM, "Eric Day" <eday at oddments.org> wrote:
>>>>>> 
>>>>>>> I've summarized the original email and added more sections for
>>>> review
>>>>>>> and discussion here:
>>>>>>> 
>>>>>>> http://wiki.openstack.org/QueueService
>>>>>>> 
>>>>>>> In particular there are details on the various components of the
>>>>>>> queue service, a draft at what the REST operations will look like,
>>>>>>> and a couple brief examples.
>>>>>>> 
>>>>>>> Please let me know if any clarification is needed and I'll update
>>>>>>> the wiki. Feedback and discussion on use cases and what you think
>>>>>>> the service should look like is very appreciated.
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> -Eric
>>>>>>> 
>>>>>>> On Mon, Feb 14, 2011 at 09:51:42AM -0800, Eric Day wrote:
>>>>>>>> Hi everyone,
>>>>>>>> 
>>>>>>>> When looking at other services to include as part of OpenStack,
>>>> the
>>>>>>>> first that comes to mind for many is a queue. A queue service can
>>>>>>>> not only provide a useful public cloud service, but can also
>>>> provide
>>>>>>>> one of the building blocks for other services. I've been leading
>>>> an
>>>>>>>> effort to research and gather requirements for a queue service
>>>> and
>>>>>>>> I'd like to share the current state and get community feedback. I
>>>>>>>> expect real development to begin very soon, and would also like
>>>> to
>>>>>>>> identify developers who will have time to dedicate to this
>>>> project.
>>>>>>>> 
>>>>>>>> I'd like to note this is not an official OpenStack project yet.
>>>> The
>>>>>>>> intention is once we have the community support and a simple
>>>>>>>> implementation, we will submit the project to the OpenStack
>>>> Project
>>>>>>>> Oversight Committee for approval.
>>>>>>>> 
>>>>>>>> The reason we are initiating our own project vs using an existing
>>>> one
>>>>>>>> is due to simplicity, modularity, and scale. Also, very few (if
>>>> any)
>>>>>>>> existing queue systems out there were built with multi-tenant
>>>> cloud
>>>>>>>> use cases in mind. Very few also have a simple and extensible
>>>> REST
>>>>>>>> API. There are possible solutions to build an AMQP based service,
>>>>>>>> but AMQP brings complexity and a protocol not optimized for high
>>>>>>>> latency and intermittent connectivity.
>>>>>>>> 
>>>>>>>> The primary goals of the queue service are:
>>>>>>>> 
>>>>>>>> * Simple - Think simple REST based queues for most use cases.
>>>> Easy
>>>>>>>>  to access and use from any language. It should not require much
>>>>>>>>  setup, if any, before you can start pushing messages into it.
>>>>>>>> 
>>>>>>>> * Modular API - Initially we'll focus on a simple REST API,
>>>>>>>>  but this will not in any way be a first-class API. It should be
>>>>>>>>  possible to add other protocols (AMQP, protocol buffers,
>>>> Gearman,
>>>>>>>>  etc) for other use cases. Note that the internal service API
>>>> will
>>>>>>>>  not always provide a 1-1 mapping with the external API, so some
>>>>>>>>  features with advanced protocols may be unavailable.
>>>>>>>> 
>>>>>>>> * Fast - Since this will act as a building block for other
>>>> services
>>>>>>>>  that my drive heavy throughput, performance will have a focus.
>>>> This
>>>>>>>>  mostly comes down to implementation language and how clients
>>>> and
>>>>>>>>  workers interact with the broker to reduce network chatter.
>>>>>>>> 
>>>>>>>> * Multi-tenant - Support multiple accounts for the service, and
>>>> since
>>>>>>>>  this will also be a public service for some deployments,
>>>> protect
>>>>>>>>  against potentially malicious users.
>>>>>>>> 
>>>>>>>> * Persistent - Allow messages to optionally be persistent. For
>>>>>>>>  protocols that can support it, this can be an optional flag
>>>> while
>>>>>>>>  the message is submitted. The persistent storage should also be
>>>>>>>>  modular so we can test various data stores and accommodate
>>>>>> different
>>>>>>>>  deployment options.
>>>>>>>> 
>>>>>>>> * Zones and locality awareness - As we've been discussing in
>>>> other
>>>>>>>>  threads, locality in cloud services is an important feature.
>>>> When
>>>>>>>>  dealing with where messages should be processed, we need to
>>>> have
>>>>>>>>  location awareness to process data where it exists to reduce
>>>>>> network
>>>>>>>>  overhead and processing time.
>>>>>>>> 
>>>>>>>> Before diving down into implementation details, I would like to
>>>> hear
>>>>>>>> what folks have to say about the initial requirements above. Once
>>>>>>>> there is something along the lines of agreement, I'll be sending
>>>> out
>>>>>>>> other topics for discussion dealing with implementation.
>>>>>>>> 
>>>>>>>> I'm looking forward to your feedback. Thanks!
>>>>>>>> 
>>>>>>>> -Eric
>>>>>>> 
>>>>>>> _______________________________________________
>>>>>>> Mailing list: https://launchpad.net/~openstack
>>>>>>> Post to     : openstack at lists.launchpad.net
>>>>>>> Unsubscribe : https://launchpad.net/~openstack
>>>>>>> More help   : https://help.launchpad.net/ListHelp
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Confidentiality Notice: This e-mail message (including any attached
>>>> or
>>>>>> embedded documents) is intended for the exclusive and confidential
>>>> use
>>>>>> of the
>>>>>> individual or entity to which this message is addressed, and unless
>>>>>> otherwise
>>>>>> expressly indicated, is confidential and privileged information of
>>>>>> Rackspace.
>>>>>> Any dissemination, distribution or copying of the enclosed material
>>>> is
>>>>>> prohibited.
>>>>>> If you receive this transmission in error, please notify us
>>>> immediately
>>>>>> by e-mail
>>>>>> at abuse at rackspace.com, and delete the original message.
>>>>>> Your cooperation is appreciated.
>>>> 
>>>> 
>>>> 
>>>> Confidentiality Notice: This e-mail message (including any attached or
>>>> embedded documents) is intended for the exclusive and confidential use
>>>> of the
>>>> individual or entity to which this message is addressed, and unless
>>>> otherwise
>>>> expressly indicated, is confidential and privileged information of
>>>> Rackspace. 
>>>> Any dissemination, distribution or copying of the enclosed material is
>>>> prohibited.
>>>> If you receive this transmission in error, please notify us immediately
>>>> by e-mail
>>>> at abuse at rackspace.com, and delete the original message.
>>>> Your cooperation is appreciated.
>> 
>> 
>> 
>> Confidentiality Notice: This e-mail message (including any attached or
>> embedded documents) is intended for the exclusive and confidential use of
>> the
>> individual or entity to which this message is addressed, and unless
>> otherwise
>> expressly indicated, is confidential and privileged information of
>> Rackspace.
>> Any dissemination, distribution or copying of the enclosed material is
>> prohibited.
>> If you receive this transmission in error, please notify us immediately
>> by e-mail
>> at abuse at rackspace.com, and delete the original message.
>> Your cooperation is appreciated.
>> 
>> 
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openstack
>> Post to     : openstack at lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
> 
> 
> _______________________________________________
> Mailing list: https://launchpad.net/~openstack
> Post to     : openstack at lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp





More information about the Openstack mailing list