<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:odc="urn:schemas-microsoft-com:office:odc" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:rtc="http://microsoft.com/officenet/conferencing" xmlns:D="DAV:" xmlns:Repl="http://schemas.microsoft.com/repl/" xmlns:mt="http://schemas.microsoft.com/sharepoint/soap/meetings/" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ppda="http://www.passport.com/NameSpace.xsd" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcs="http://schemas.microsoft.com/data/udc/soap" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:udcp2p="http://schemas.microsoft.com/data/udc/parttopart" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:dsss="http://schemas.microsoft.com/office/2006/digsig-setup" xmlns:dssi="http://schemas.microsoft.com/office/2006/digsig" xmlns:mdssi="http://schemas.openxmlformats.org/package/2006/digital-signature" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:spwp="http://microsoft.com/sharepoint/webpartpages" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:pptsl="http://schemas.microsoft.com/sharepoint/soap/SlideLibrary/" xmlns:spsl="http://microsoft.com/webservices/SharePointPortalServer/PublishedLinksService" xmlns:Z="urn:schemas-microsoft-com:" xmlns:st="" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-GB link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Raphael,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Could you tell us more about StormMQ? What do you do, how much
of your software is open-source, how might it fit into the OpenStack ecosystem
(both from open-source and proprietary points-of-view)?<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>I admit to my shame that I know nothing about your company, but
it certainly sounds like you know what you’re talking about, so I’m curious to learn
more!<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Thanks,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Ewan.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'>
<p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:
"Tahoma","sans-serif"'>From:</span></b><span lang=EN-US style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'>
openstack-bounces+ewan.mellor=citrix.com@lists.launchpad.net
[mailto:openstack-bounces+ewan.mellor=citrix.com@lists.launchpad.net] <b>On
Behalf Of </b>Raphael Cohn<br>
<b>Sent:</b> 28 February 2011 02:02<br>
<b>To:</b> Eric Day<br>
<b>Cc:</b> openstack@lists.launchpad.net<br>
<b>Subject:</b> Re: [Openstack] Queue Service, next steps<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'>Eric,<br>
<br>
Thank you.<br>
<br>
You raise lots of interest points. In no particular order:-<br>
<br>
AMQP Observations<br>
Your comments about AMQP seem to mostly be appropriate for one of the older
versions, eg 0-8, and I don't think they particularly apply to later versions,
eg 1-0. AMQP 0-8 did have some issues that didn't always make it an optimal
choice. For instance:-<br>
- Connection Handshaking. This is pipelined in AMQP 1.0, so a connection can
open, send messages and disconnect without ever receiving a counter party's
ack.<br>
- Reliability. When we implemented Smith's telemetry solution - the largest in
use in the world for Electric Vehicles - we experimented with HTTPS based
messaging over unreliable links using real hardware and real networks, as, like
you, we thought it would be better for short-lived connections. AMQP works
better, hence we never put a REST API into production. The final decider was
the difficulty of ensuring at-least-once with HTTP.<br>
- Exchanges, etc are no longer part of the spec; queues can be transient, with
configurable transience and timeouts (eg destroy it 10s after the last message
was retrieved)<br>
- Configuration is part of the act of sending messages, not separate, eg open,
send to new queue, etc<br>
- There's nothing to stop you using a connection-less protocol with AMQP, such
as UDP (or perhaps UDT or ENET, which work ontop of UDP). You would simply need
to send heartbeats every now and again to make sure the connection was kept
alive.<br>
<br>
Using HTTP<br>
Whilst you can always put any asynchronous protocol over a synchronous one, it
doesn't always work out too well. For example, starting on such an
approach means that any 'kernel' will be optimised for 'pulling' from a queue,
when an efficient queue serve handling tens of thousands of connections needs
to be able to 'push' incoming messages, after filtering, to their destinations.
Pushing it all into the HTTP request is a sensible approach for simple
req-response protocols, but it's going to put a heavy burden onto your queue
server.<br>
RTT: This is almost irrelevant once you decide to use TLS. TLS set up and tear
down, essential for most cloud operations, is far more inefficient than any
protocol it tunnels. And anyone sending messages without encryption should be
shot. It's not acceptable to send other people's data unsecured anymore
(indeed, if it ever was).<br>
201 Created, etc: What happens to you message if your TCP connection dies mid
reply? How do you know if your message was queued? Is there a reconcillation
API?<br>
<br>
Of course, some of these concerns could be addressed with HTTP sessions or
cookies, but that's quite nasty to use in most environments. Fundamentally, if
your wish is to support languages such as PHP with a messaging layer, then HTTP
initially seems to be the way to go. The reality is that any TCP based approach
is inappropriate here, because opening a TCP connection on the back of a TCP
connection is very weak. The right solution is to use connection caching -
people have done it with databases for years for this reason - but some of
these, erm, web languages, make such an approach too hard. Hopefully the growth
of sensible back ends like node.js will make this a thing of the past. AMQP has
internally multiplexed sessions (virtual connections if you will), so a PHP
runtime, say, could open just one AMQP connection - and assign each incoming
request to one of the available sessions. With 65,536 sessions available, only
the most incredible PHP server would need them all at once, given that most PHP
code falls over if 3 people connect... In practice, the right place then to
open a messaging connection with the sorts of web apps these languages are used
for is in the browser itself - a job for which WebSockets, and not HTTP, would
seem the right solution. And AMQP is rather well suited to WebSockets.<br>
<br>
WebSockets<br>
These are definitely cloud-friendly... and designed to replace HTTP approaches
for these sorts of problems.<br>
<br>
Transient Queues<br>
Oh, these are really easy to do. But distributed hashing isn't. It's actually a
really interesting problem for queuing, and one we had to address with StormMQ.
Intriguingly, in low usage situations, it actually makes message receipt
non-deterministic and potentially unordered; getting this 'right' depends if
you in err in favour of at-most-once messaging or at-least-once messsaging.<br>
<br>
Raph<br>
<br clear=all>
Raphael Cohn<br>
Managing Director<br>
raphael.cohn@StormMQ.com<br>
StormMQ Limited<br>
<br>
UK Office: <br>
Gateshead int'l Business Centre, Mulgrave Terrace, Gateshead, NE8 1AN, United
Kingdom<br>
Telephone: +44 845 3712 567<br>
<br>
Registered office:<br>
78 Broomfield Road, Chelmsford, Essex, CM1 1SS, United Kingdom<br>
StormMQ Limited is Registered in England and Wales under Company Number
07175657<br>
StormMQ.com<br>
<br>
<o:p></o:p></p>
<div>
<p class=MsoNormal>On 27 February 2011 19:46, Eric Day <<a
href="mailto:eday@oddments.org">eday@oddments.org</a>> wrote:<o:p></o:p></p>
<p class=MsoNormal>Hi Raphael,<o:p></o:p></p>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
On Sun, Feb 27, 2011 at 11:18:35AM +0000, Raphael Cohn wrote:<br>
> OpenStack's QueueService seems very interesting. As we have
an existing<br>
> message queue implementation, we'd be happy to help you guys
out. We're<br>
> about making messaging cloud-scale, so that everyone
benefits.<o:p></o:p></p>
</div>
<p class=MsoNormal>Thank you! We're certainly looking to include as many
community members<br>
as we can to ensure this is a successful project. You expertise and<br>
participation would be very much appreciated!<o:p></o:p></p>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
> However, it worries us that you're planning to implement a
REST API for<br>
> messaging. Message queuing is fundamentally asynchronous;
this is one of<br>
> the reasons StormMQ got started, as we found that approaches
that use it<br>
> (eg SQS) suffer from some major weaknesses:-<br>
> - They're too slow;<br>
> - They can't handle sustained volumes<br>
> - Higher-level needs, eg fanout, selective pub-sub and
transactions, are<br>
> an awkward, if not impossible, fit<o:p></o:p></p>
</div>
<p class=MsoNormal>I certainly agree, HTTP is not an ideal protocol for
high-performance<br>
messaging. Some features may be awkward in HTTP, but almost anything<br>
is possible. As you'll note on the queue service specification page,<br>
a pluggable protocol is one of the main requirements. The REST API<br>
is the first since this is the easiest protocol for most folks to<br>
understand and get involved with, it is by no means the primary or a<br>
first-class protocol. For example, I mention other binary protocols<br>
to look at implementing for higher performance once we get the REST<br>
API off the ground.<br>
<br>
HTTP though, if done correctly (pipelining, binary content-types,<br>
...), can provide decent throughput that is sufficient for a wide<br>
range of applications. It will always be restricted by the plain-text<br>
request/header envelope, and this is where binary protocols will excel.<br>
<br>
Also, not all users and use cases of the queue service will need<br>
to prioritize on high throughput. The overhead of the HTTP protocol<br>
parsing may be insignificant for some, and instead the accessibility<br>
of the service via HTTP in their environment (web apps, browser,<br>
etc.) may be much more important than high throughput. Accessibility,<br>
especially now in a very RESTy web/cloud world, is very import.<o:p></o:p></p>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
> There are a hoard of technical reasons why HTTP, superb as it
is for<br>
> request-response architectures, makes a poor backbone for
messaging (some<br>
> of the team behind StormMQ implemented one of the first
banking-scale REST<br>
> architectures).<br>
><br>
> For example, implementations that need to send or consume
lots of data,<br>
> and are only interested in a subset whose filter criteria
changes over<br>
> time. Syslogging, for example. Imagine a dynamic cloud, where
servers come<br>
> and go - and centralised logging systems and alerts need no
configuration,<br>
> because they use queuing. Under load (eg hack attempts on
your server<br>
> firewalls generate 1000s of log messages) it mustn't fail,
just go a bit<br>
> slower. StormMQ use AMQP internally for our own log
management for that<br>
> reason.<o:p></o:p></p>
</div>
<p class=MsoNormal>Understood, and much of this can be accomplished with
horizontally<br>
scaling architectures. As I touched on before and mentioned on<br>
the wiki, HTTP is only one interface in. The internal communication<br>
protocol for scaling out zones and clusters will not be HTTP long term,<br>
and instead a much more efficient, async, and binary protocol. My<br>
current thought is to use Google protocol buffers or Avro for this,<br>
but this is up in the air (something we won't get to for at least a<br>
couple months). Since we're using Erlang, we may even use the native<br>
Erlang message passing if we're on a trusted network.<o:p></o:p></p>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
> First up, AMQP isn't actually very complex at the level of an
application<br>
> developer. Indeed, with a good library (like ours) it's
trivially easy.<o:p></o:p></p>
</div>
<p class=MsoNormal>Agreed, there are some great AMQP libraries out there that
make it<br>
seamless, but there are also some that do not. This wasn't my concern<br>
with the complexity comment though.<o:p></o:p></p>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
> The apparent complexity comes becomes of unfamiliarity, both
with concepts<br>
> and with use; no different to HTTP when it first came in (and
we saw a<br>
> plethora of weird ways of using it and misunderstood criteria
for headers,<br>
> etc). AMQP's highly suited to high-latency, unreliable links.
That's why<br>
> Smith Electric vehicles use it to connect all their delivery
trucks using<br>
> dodgy 3G links - and still gather 10,000s of items of data a
second. The<br>
> AMQP protocol, particularly 1.0, make it's extremely clear
how and when to<br>
> recover from failure. Indeed, AMQP's approach is failure
happens - so deal<br>
> with it. HTTP on the other hand, has no such level of
transactionality.<o:p></o:p></p>
</div>
<p class=MsoNormal>For the complexity concern, my main point is that in order
to use<br>
a queue, you need a channel, exchange, queue, and a binding between<br>
an exchange/queue. This can be made fairly trivial by libraries you<br>
mentioned, but there are a lot of objects and relationships to keep<br>
in sync in a distributed system. The OpenStack queue service takes a<br>
fundamentally different approach and requires no queue setup before<br>
you can put a message into it. Queues (and accounts) are transient,<br>
when a message is inserted into a queue, or when a consumer is<br>
waiting on queue, it comes into existence. When the queue is empty,<br>
it disappears. This allows you to easily create temporary queues<br>
without worry of race conditions between producers and consumers.<br>
<br>
As for my comment on AMQP's suitability for highly-latent or<br>
unreliable links, it is primary directed towards the 7-way handshake<br>
for consumers, and 4-way handshake for producers (both on top of one<br>
RTT for the TCP handshake). Once these connections are established the<br>
protocol is very efficient, but this doesn't help with unreliable links<br>
or environments where persistent connections are hard or impossible<br>
to maintain. AMQP will certainly work in these environments, but it<br>
seems it is much more suited for reliable links where the handshake<br>
isn't required as often.<br>
<br>
With the proposed OpenStack queue service REST API, there will only<br>
be one RTT for both producers and consumers (on top of one RTT for<br>
TCP). A producer will make a PUT request with a 201 Created response. A<br>
consumer is a GET or POST with response body. All authentication,<br>
queue destination, and other metadata will be included in the request,<br>
rather than building up a stateful channel through the handshake.<br>
<br>
Cloud, and especially module, use cases bring much higher latency than<br>
is typically seen in clustered environments. Short-lived connections<br>
are always possible depending on the developer or environment (not<br>
just due to unreliable links, for example connection caching may be<br>
difficult or impossible). This is why an emphasis was put on stateless<br>
communication with minimal round trips.<o:p></o:p></p>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
> Second up, more importantly, StormMQ do not provide a REST
API as an<br>
> alternative to AMQP. It's to provide features that are
nothing to do with<br>
> message queuing - dynamically slicing up your cloud, for
instance, or<br>
> managing environments to allow exact reproduceability or
checking in to<br>
> source control your config. We'd be interested in providing a
REST API if<br>
> there's the demand. AMQP does support multi-tenancy - we do
it.<o:p></o:p></p>
</div>
<p class=MsoNormal>We plan to address these issues with this project,
especially<br>
multi-tenancy and multi-zone interaction. We need this to not only<br>
handle the simple use cases, but to also run a public cloud service.<o:p></o:p></p>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
> To assist, pragamatically, we'd like to donate as open source
our upcoming<br>
> C and Java clients for AMQP 1.0, and help sponsor Python,
Perl, PHP and<br>
> Ruby ones off the C code, so that there is as wide as
possible opportunity<br>
> for people to use messaging.<o:p></o:p></p>
</div>
<p class=MsoNormal>Thanks! Before being able to fully leverage these, we'll
also need an<br>
AMQP binding, which to be honest I've given very little thought. Once<br>
we have a solid queue "kernel" this will be easier, but I'm certainly<br>
keeping AMQP semantics in mind. We are also using RabbitMQ for the<br>
Nova project using the carrot Python module. It might be interesting<br>
to see how your clients compare and if they may benefit that project.<o:p></o:p></p>
<div>
<p class=MsoNormal style='margin-bottom:12.0pt'><br>
> I'd strongly encourage you to get involved in the AMQP
working group so if<br>
> there's needs that are not met by AMQP, they can be
addressed. The working<br>
> group is really keen to encourage an open, widely adopted
standard for<br>
> AMQP; they'd like it to be the HTTP of messaging. Many of the
features I<br>
> see proposed for OpenStack are features in AMQP - and AMQP
has spent a lot<br>
> of time working out the kinks in edge cases and making sure
they'd work<br>
> with the legacy - JMS, TIBCO and the like.<o:p></o:p></p>
</div>
<p class=MsoNormal>I'll certainly consider it, but I'd first like to get a
functional<br>
service up and running to see how these ideas (distributed hashing,<br>
stateless, transient queues) hold up and then we can see what features,<br>
if any, would make sense as an AMQP proposal.<br>
<br>
Thanks again for you input! I'm looking forward to further discussion<br>
and StormMQ's participation.<br>
<span style='color:#888888'><br>
-Eric</span><o:p></o:p></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</div>
</body>
</html>