<div dir="ltr"><div><div>Kurt Griffiths,<br><br>Thanks for detailed explanation. Is there a comparison between Marconi and existing message brokers anywhere that you can point me out?<br></div>I can see how your examples can be implemented using other brokers like RabbitMQ. So why there is a need another broker? And what is wrong with currently deployed RabbitMQ that most of OpenStack services are using (typically via oslo.messaging RPC)?<br>
</div><br></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Mar 19, 2014 at 4:00 AM, Kurt Griffiths <span dir="ltr"><<a href="mailto:kurt.griffiths@rackspace.com" target="_blank">kurt.griffiths@rackspace.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">I think we can agree that a data-plane API only makes sense if it is<br>
useful to a large number of web and mobile developers deploying their apps<br>
on OpenStack. Also, it only makes sense if it is cost-effective and<br>
scalable for operators who wish to deploy such a service.<br>
<br>
Marconi was born of practical experience and direct interaction with<br>
prospective users. When Marconi was kicked off a few summits ago, the<br>
community was looking for a multi-tenant messaging service to round out<br>
the OpenStack portfolio. Users were asking operators for something easier<br>
to work with and more web-friendly than established options such as AMQP.<br>
<br>
To that end, we started drafting an HTTP-based API specification that<br>
would afford several different messaging patterns, in order to support the<br>
use cases that users were bringing to the table. We did this completely in<br>
the open, and received lots of input from prospective users familiar with<br>
a variety of message broker solutions, including more “cloudy” ones like<br>
SQS and Iron.io.<br>
<br>
The resulting design was a hybrid that supported what you might call<br>
“claim-based” semantics ala SQS and feed-based semantics ala RSS.<br>
Application developers liked the idea of being able to use one or the<br>
other, or combine them to come up with new patterns according to their<br>
needs. For example:<br>
<br>
1. A video app can use Marconi to feed a worker pool of transcoders. When<br>
a video is uploaded, it is stored in Swift and a job message is posted to<br>
Marconi. Then, a worker claims the job and begins work on it. If the<br>
worker crashes, the claim expires and the message becomes available to be<br>
claimed by a different worker. Once the worker is finished with the job,<br>
it deletes the message so that another worker will not process it, and<br>
claims another message. Note that workers never “list” messages in this<br>
use case; those endpoints in the API are simply ignored.<br>
<br>
2. A backup service can use Marconi to communicate with hundreds of<br>
thousands of backup agents running on customers' machines. Since Marconi<br>
queues are extremely light-weight, the service can create a different<br>
queue for each agent, and additional queues to broadcast messages to all<br>
the agents associated with a single customer. In this last scenario, the<br>
service would post a message to a single queue and the agents would simply<br>
list the messages on that queue, and everyone would get the same message.<br>
This messaging pattern is emergent, and requires no special routing setup<br>
in advance from one queue to another.<br>
<br>
3. A metering service for an Internet application can use Marconi to<br>
aggregate usage data from a number of web heads. Each web head collects<br>
several minutes of data, then posts it to Marconi. A worker periodically<br>
claims the messages off the queue, performs the final aggregation and<br>
processing, and stores the results in a DB. So far, this messaging pattern<br>
is very much like example #1, above. However, since Marconi’s API also<br>
affords the observer pattern via listing semantics, the metering service<br>
could run an auditor that logs the messages as they go through the queue<br>
in order to provide extremely valuable data for diagnosing problems in the<br>
aggregated data.<br>
<br>
Users are excited about what Marconi offers today, and we are continuing<br>
to evolve the API based on their feedback.<br>
<br>
Of course, app developers aren’t the only audience Marconi needs to serve.<br>
Operators want something that is cost-effective, scales, and is<br>
customizable for the unique needs of their target market.<br>
<br>
While Marconi has plenty of room to improve (who doesn’t?), here is where<br>
the project currently stands in these areas:<br>
<br>
1. Customizable. Marconi transport and storage drivers can be swapped out,<br>
and messages can be manipulated in-flight with custom filter drivers.<br>
Currently we have MongoDB and SQLAlchemy drivers, and are exploring Redis<br>
and AMQP brokers. Now, the v1.0 API does impose some constraints on the<br>
backend in order to support the use cases mentioned earlier. For example,<br>
an AMQP backend would only be able to support a subset of the current API.<br>
Operators occasionally ask about AMQP broker support, in particular, and<br>
we are exploring ways to evolve the API in order to support that.<br>
<br>
2. Scalable. Operators can use Marconi’s HTTP transport to leverage their<br>
existing infrastructure and expertise in scaling out web heads. When it<br>
comes to the backend, for small deployments with minimal throughput needs,<br>
we are providing a SQLAlchemy driver as a non-AGPL alternative to MongoDB.<br>
For large-scale production deployments, we currently provide the MongoDB<br>
driver and will likely add Redis as another option (there is already a POC<br>
driver). And, of course, operators can provide drivers for NewSQL<br>
databases, such as VelocityDB, that are very fast and scale extremely<br>
well. In Marconi, every queue can be associated with a different backend<br>
cluster. This allows operators to scale both up and out, according to what<br>
is most cost-effective for them. Marconi's app-level sharding is currently<br>
done using a lookup table to provide for maximum operator control over<br>
placement, but I personally think it would be great to see this opened up<br>
so that we can swap in other types of drivers, such as one based on hash<br>
rings (TBD).<br>
<br>
3. Cost-effective. The Marconi team has done a lot of work to (1) provide<br>
several dimensions for scaling deployments that can be used according to<br>
what is most cost-effective for a given use case, and (2) make the Marconi<br>
service as efficient as possible, including time spent optimizing the<br>
transport layer (using Falcon in lieu of Pecan, reducing the work that the<br>
request handlers do, etc.), and tuning the MongoDB storage driver (the<br>
SQLAlchemy driver is newer and we haven’t had the chance to tune it yet,<br>
but are planning to do so during Juno). Turnaround on requests is in the<br>
low ms range (including dealing with HTTP), not the usec range, but that<br>
works perfectly well for a large class of applications. We’ve been<br>
benchmarking with Tsung for quite a while now, and we are working on<br>
making the raw data more accessible to folks outside our team. I’ll try to<br>
get some of the latest data up on the wiki this week.<br>
<br>
Marconi was originally incubated because the community believed developers<br>
building their apps on top of OpenStack were looking for this kind of<br>
service, and it was a big missing gap in our portfolio. Since that time,<br>
the team has worked hard to fill that gap.<br>
<span class="HOEnZb"><font color="#888888"><br>
Kurt<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div dir="ltr"><span style="border-collapse:separate;color:rgb(0,0,0);font-family:'Times New Roman';font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><span style="font-family:arial;font-size:small">Sincerely yours<br>
Stanislav (Stan) Lagun<br>Senior Developer<br>Mirantis</span></span><br><span style="border-collapse:separate;color:rgb(0,0,0);font-family:'Times New Roman';font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;font-size:medium"><span style="font-family:arial;font-size:small"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"" lang="EN-US">35b/3, Vorontsovskaya
St.</span><br>Moscow, Russia<br>Skype: stanlagun<br><a href="http://www.mirantis.com/" target="_blank">www.mirantis.com</a><br><a href="mailto:slagun@mirantis.com" target="_blank">slagun@mirantis.com</a></span></span></div>
</div>