[openstack-dev] [Marconi] Kafka support and high throughput
flavio at redhat.com
Thu Jun 5 07:02:11 UTC 2014
On 04/06/14 14:58 +0000, Hochmuth, Roland M wrote:
>Hi Flavio, In your discussions around a developing Kafka plugin for
>Marconi would that be potentially be done by adding a Kafka transport to
>oslo.messaging? That is something that I'm very interested in for the
>monitoring as a service project I'm working on.
No, oslo.messaging is a different project and Marconi doesn't rely on
>On 6/4/14, 3:06 AM, "Flavio Percoco" <flavio at redhat.com> wrote:
>>On 02/06/14 07:52 -0700, Keith Newstadt wrote:
>>>Thanks for the responses Flavio, Roland.
>>>Some background on why I'm asking: we're using Kafka as the message
>>>queue for a stream processing service we're building, which we're
>>>delivering to our internal customers as a service along with OpenStack.
>>>We're considering building a high throughput ingest API to get the
>>>clients' data streams into the stream processing service. It occurs to
>>>me that this API is simply a messaging API, and so I'm wondering if we
>>>should consider building this high throughput API as part of the Marconi
>>>Has this topic come up in the Marconi team's discussions, and would it
>>>fit into the vision of the Marconi roadmap?
>>Yes it has and I'm happy to see this coming up in the ML, thanks.
>>Some things that we're considering in order to have a more flexible
>>architecture that will support a higher throughput are:
>>- Queue Flavors (Terrible name). This is for marconi what flavors are
>> for Nova. It basically defines a set of properties that will belong
>> to a queue. Some of those properties may be related to the messages
>> lifetime or the storage capabilities (in-memory, freaking fast,
>> durable, etc). This is yet to be done.
>>- 2 new drivers (AMQP, redis). The former adds support to brokers and
>> the later to well, redis, which brings in support for in-memory
>> queues. Work In Progress.
>>- A new transport. This is something we've discussed but we haven't
>> reached an agreement yet on when this should be done nor what it
>> should be based on. The gist of this feature is adding support for
>> another protocol that can serve Marconi's API alongside the HTTP
>> one. We've considered TCP and websocket so far. The former is
>> perfect for lower level communications without the HTTP overhead
>> whereas the later is useful for web apps.
>>That said. A Kafka plugin is something we heard a lot about at the
>>summit and we've discussed it a bit. I'd love to see that happening as
>>an external plugin for now. There's no need to wait for the rest to
>>I'm more than happy to help with guidance and support on the repo
>>creation, driver structure etc.
>>>keith_newstadt at symantec.com
>>>Date: Sun, 1 Jun 2014 15:01:40 +0000
>>>From: "Hochmuth, Roland M" <roland.hochmuth at hp.com>
>>>To: OpenStack List <openstack-dev at lists.openstack.org>
>>>Subject: Re: [openstack-dev] [Marconi] Kafka support and high
>>>Message-ID: <CFAE6524.762DA%roland.hochmuth at hp.com>
>>>Content-Type: text/plain; charset="us-ascii"
>>>There are some folks in HP evaluating different messaging technologies
>>>Marconi, such as RabbitMQ and Kafka. I'll ping them and maybe they can
>>>On a related note, the Monitoring as a Service solution we are working
>>>on uses Kafka. This was just open-sourced at,
>>>and will be moving over to StackForge starting next week. The
>>>I haven't really looked at Marconi. If you are interested in
>>>throughput, low latency, durability, scale and fault-tolerance Kafka
>>>seems like a great choice.
>>>It has been also pointed out from various sources that possibly Kafka
>>>could be another oslo.messaging transport. Are you looking into that as
>>>that would be very interesting to me and something that is on my task
>>>list that I haven't gotten to yet.
>>>On 5/30/14, 7:03 AM, "Keith Newstadt" <keith_newstadt at symantec.com>
>>>>Has anyone given thought to using Kafka to back Marconi? And has there
>>>>been discussion about adding high throughput APIs to Marconi.
>>>>We're looking at providing Kafka as a messaging service for our
>>>>customers, in a scenario where throughput is a priority. We've had good
>>>>luck using both streaming HTTP interfaces and long poll interfaces to
>>>>high throughput for other web services we've built. Would this use case
>>>>be appropriate in the context of the Marconi roadmap?
>>>>keith_newstadt at symantec.com
>>>Cloud Services Architect
>>>Cloud Platform Engineering
>>>Office: (781) 530-2299 Mobile: (617) 513-1321
>>>Email: keith_newstadt at symantec.com
>>>This message (including any attachments) is intended only for the use of
>>>the individual or entity to which it is addressed and may contain
>>>information that is non-public, proprietary, privileged, confidential,
>>>and exempt from disclosure under applicable law or may constitute as
>>>attorney work product. If you are not the intended recipient, you are
>>>hereby notified that any use, dissemination, distribution, or copying of
>>>this communication is strictly prohibited. If you have received this
>>>communication in error, notify us immediately by telephone and (i)
>>>destroy this message if a facsimile or (ii) delete this message
>>>immediately if this is an electronic communication.
>>>OpenStack-dev mailing list
>>>OpenStack-dev at lists.openstack.org
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 819 bytes
Desc: not available
More information about the OpenStack-dev