Open Stack

Hi Flavio, In your discussions around a developing Kafka plugin for
Marconi would that be potentially be done by adding a Kafka transport to
oslo.messaging? That is something that I'm very interested in for the
monitoring as a service project I'm working on.

Thanks --Roland


On 6/4/14, 3:06 AM, "Flavio Percoco" <flavio at redhat.com> wrote:

>On 02/06/14 07:52 -0700, Keith Newstadt wrote:
>>Thanks for the responses Flavio, Roland.
>>
>>Some background on why I'm asking:  we're using Kafka as the message
>>queue for a stream processing service we're building, which we're
>>delivering to our internal customers as a service along with OpenStack.
>>We're considering building a high throughput ingest API to get the
>>clients' data streams into the stream processing service.  It occurs to
>>me that this API is simply a messaging API, and so I'm wondering if we
>>should consider building this high throughput API as part of the Marconi
>>project.
>>
>>Has this topic come up in the Marconi team's discussions, and would it
>>fit into the vision of the Marconi roadmap?
>
>Yes it has and I'm happy to see this coming up in the ML, thanks.
>
>Some things that we're considering in order to have a more flexible
>architecture that will support a higher throughput are:
>
>- Queue Flavors (Terrible name). This is for marconi what flavors are
>  for Nova. It basically defines a set of properties that will belong
>  to a queue. Some of those properties may be related to the messages
>  lifetime or the storage capabilities (in-memory, freaking fast,
>  durable, etc). This is yet to be done.
>
>- 2 new drivers (AMQP, redis). The former adds support to brokers and
>  the later to well, redis, which brings in support for in-memory
>  queues. Work In Progress.
>
>- A new transport. This is something we've discussed but we haven't
>  reached an agreement yet on when this should be done nor what it
>  should be based on. The gist of this feature is adding support for
>  another protocol that can serve Marconi's API alongside the HTTP
>  one. We've considered TCP and websocket so far. The former is
>  perfect for lower level communications without the HTTP overhead
>  whereas the later is useful for web apps.
>
>That said. A Kafka plugin is something we heard a lot about at the
>summit and we've discussed it a bit. I'd love to see that happening as
>an external plugin for now. There's no need to wait for the rest to
>happen.
>
>I'm more than happy to help with guidance and support on the repo
>creation, driver structure etc.
>
>Cheers,
>Flavio
>
>>
>>Thanks,
>>Keith Newstadt
>>keith_newstadt at symantec.com
>>@knewstadt
>>
>>
>>Date: Sun, 1 Jun 2014 15:01:40 +0000
>>From: "Hochmuth, Roland M" <roland.hochmuth at hp.com>
>>To: OpenStack List <openstack-dev at lists.openstack.org>
>>Subject: Re: [openstack-dev] [Marconi] Kafka support and high
>>	throughput
>>Message-ID: <CFAE6524.762DA%roland.hochmuth at hp.com>
>>Content-Type: text/plain; charset="us-ascii"
>>
>>There are some folks in HP evaluating different messaging technologies
>>for
>>Marconi, such as RabbitMQ and Kafka. I'll ping them and maybe they can
>>share
>>some information.
>>
>>On a related note, the Monitoring as a Service solution we are working
>>on uses Kafka. This was just open-sourced at,
>>https://github.com/hpcloud-mon,
>>and will be moving over to StackForge starting next week. The
>>architecture
>>is at,
>>https://github.com/hpcloud-mon/mon-arch.
>>
>>I haven't really looked at Marconi. If you are interested in
>>throughput, low latency, durability, scale and fault-tolerance Kafka
>>seems like a great choice.
>>
>>It has been also pointed out from various sources that possibly Kafka
>>could be another oslo.messaging transport. Are you looking into that as
>>that would be very interesting to me and something that is on my task
>>list that I haven't gotten to yet.
>>
>>
>>On 5/30/14, 7:03 AM, "Keith Newstadt" <keith_newstadt at symantec.com>
>>wrote:
>>
>>>Has anyone given thought to using Kafka to back Marconi?  And has there
>>>been discussion about adding high throughput APIs to Marconi.
>>>
>>>We're looking at providing Kafka as a messaging service for our
>>>customers, in a scenario where throughput is a priority.  We've had good
>>>luck using both streaming HTTP interfaces and long poll interfaces to
>>>get
>>>high throughput for other web services we've built.  Would this use case
>>>be appropriate in the context of the Marconi roadmap?
>>>
>>>Thanks,
>>>Keith Newstadt
>>>keith_newstadt at symantec.com
>>>
>>
>>
>>
>>
>>Keith Newstadt
>>Cloud Services Architect
>>Cloud Platform Engineering
>>Symantec Corporation
>>www.symantec.com
>>________________________________
>>
>>Office: (781) 530-2299  Mobile: (617) 513-1321
>>Email: keith_newstadt at symantec.com
>>Twitter: @knewstadt
>>________________________________
>>
>>
>>
>>This message (including any attachments) is intended only for the use of
>>the individual or entity to which it is addressed and may contain
>>information that is non-public, proprietary, privileged, confidential,
>>and exempt from disclosure under applicable law or may constitute as
>>attorney work product. If you are not the intended recipient, you are
>>hereby notified that any use, dissemination, distribution, or copying of
>>this communication is strictly prohibited. If you have received this
>>communication in error, notify us immediately by telephone and (i)
>>destroy this message if a facsimile or (ii) delete this message
>>immediately if this is an electronic communication.
>>
>>
>>
>>_______________________________________________
>>OpenStack-dev mailing list
>>OpenStack-dev at lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>-- 
>@flaper87
>Flavio Percoco