<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">Let me start by saying that I want there to be a constructive discussion around all this. I've done my best to keep my tone as non-snarky as I could while still clearly stating my concerns. I've also spent a few hours reviewing the
current code and docs. Hopefully this contribution will be beneficial in helping the discussion along.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">For what it's worth, I don't have a clear understanding of why the Marconi developer community chose to create a new queue rather than an abstraction layer on top of existing queues. While my lack of understanding there isn't a technical
objection to the project, I hope they can address this in the aforementioned FAQ. </div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">The reference storage implementation is MongoDB. AFAIK, no integrated projects require an AGPL package to be installed, and from the discussions I've been part of, that would be a show-stopper if Marconi required MongoDB. As I understand
it, this is why sqlalchemy support was required when Marconi was incubated. Saying "Marconi also supports SQLA" is disingenuous because it is a second-class citizen, with incomplete API support, is clearly not the recommended storage driver, and is going to
be unusuable at scale (I'll come back to this point in a bit).</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Let me ask this. Which back-end is tested in Marconi's CI? That is the back-end that matters right now. If that's Mongo, I think there's a problem. If it's SQLA, then I think Marconi should declare any features which SQLA doesn't support
to be optional extensions, make SQLA the default, and clearly document how to deploy Marconi at scale with a SQLA back-end.</div>
</div>
</blockquote>
<div><br>
</div>
<div>
<pre style="">"[drivers]
storage = mongodb
[drivers:storage:mongodb]
uri = mongodb://localhost:27017/marconi
</pre>
</div>
<div><br>
</div>
<div><br>
</div>
<div><a href="http://logs.openstack.org/94/81094/2/check/check-tempest-dsvm-marconi/c006285/logs/etc/marconi/marconi.conf.txt.gz" target="_blank">http://logs.openstack.org/94/81094/2/check/check-tempest-dsvm-marconi/c006285/logs/etc/marconi/marconi.conf.txt.gz</a><br>
</div>
<div> </div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>On an related note I see that marconi has no gating integration tests.</div>
<div>
<div><font color="#000000" face="Arial Unicode MS,Arial,sans-serif"><a href="https://review.openstack.org/#/c/81094/2" target="_blank">https://review.openstack.org/#/c/81094/2</a></font></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>
<div><font color="#000000" face="Arial Unicode MS,Arial,sans-serif">But then again that is documented in <a href="https://wiki.openstack.org/wiki/Marconi/Incubation/Graduation#Legal_requirements" target="_blank">https://wiki.openstack.org/wiki/Marconi/Incubation/Graduation#Legal_requirements</a></font></div>
<table style="line-height:20px;border-spacing:0px;border-collapse:collapse;max-width:100%;font-size:14.44444465637207px;background-color:rgb(249,249,249);font-family:'Arial Unicode MS',Arial,sans-serif;margin:1em 0px;border:1px solid rgb(170,170,170)">
<tbody>
<tr>
<td style="border:1px solid rgb(170,170,170);padding:0.2em">We have a devstack-gate job running and will be making it voting this week.</td>
</tr>
<tr>
</tr>
</tbody>
</table>
</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>Of the non-gating integration test job, I only see one marconi test being run: <span style="color: rgb(0, 0, 0); font-size: 9px; ">tempest.api.queuing.test_queues.TestQueues.test_create_queue</span></div>
<div> <a href="http://logs.openstack.org/94/81094/2/check/check-tempest-dsvm-marconi/c006285/logs/testr_results.html.gz">http://logs.openstack.org/94/81094/2/check/check-tempest-dsvm-marconi/c006285/logs/testr_results.html.gz</a></div>
</div>
</div>
</div>
</div>
</div>
</span>
<div>"</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<div>I have a separate thread started on the graduation gating requirements w.r.t Tempest.</div>
<div><span class="Apple-style-span">The single test we have on Tempest was a result of the one-liner requirement '</span> 'Project must have a basic devstack-gate job set up'.</div>
<div>The subsequent discussion in openstack qa meeting lead me to believe that the 'basic' job we have is good enough.</div>
<div>Please refer to the email 'Graduation Requirements + Scope of Tempest' for more details regarding this.</div>
<div><br>
</div>
<div>But that does not mean that 'the single tempest test' is all we have to verify the Marconi functionality.</div>
<div>We have had a robust test suite (unit & functional tests – with lots of positive & negative test scenarios)for a very long time in Marconi.</div>
<div>See <a href="http://logs.openstack.org/33/81033/2/check/gate-marconi-python27/35822df/testr_results.html.gz">http://logs.openstack.org/33/81033/2/check/gate-marconi-python27/35822df/testr_results.html.gz</a></div>
<div>These tests are run against a sqlite backend.</div>
<div>The gating tests have been using sqlalchemy driver ever since we have had it.</div>
<div>Hope that clarifies !</div>
<div><br>
</div>
<div>- Malini</div>
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">Then there's the db-as-a-queue antipattern, and the problems that I have seen result from this in the past... I'm not the only one in the OpenStack community with some experience scaling MySQL databases. Surely others have their own
experiences and opinions on whether a database (whether MySQL or Mongo or Postgres or ...) can be used in such a way _at_scale_ and not fall over from resource contention. I would hope that those members of the community would chime into this discussion at
some point. Perhaps they'll even disagree with me!</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">A quick look at the code around claim (which, it seems, will be the most commonly requested action) shows why this is an antipattern.</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">The MongoDB storage driver for claims requires _four_ queries just to get a message, with a serious race condition (but at least it's documented in the code) if multiple clients are claiming messages in the same queue at the same time.
For reference:</div>
<div class="gmail_extra"> <a href="https://github.com/openstack/marconi/blob/master/marconi/queues/storage/mongodb/claims.py#L119" target="_blank">
https://github.com/openstack/marconi/blob/master/marconi/queues/storage/mongodb/claims.py#L119</a><br>
</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">The SQLAlchemy storage driver is no better. It's issuing _five_ queries just to claim a message (including a query to purge all expired claims every time a new claim is created). The performance of this transaction under high load is
probably going to be bad...</div>
<div class="gmail_extra"> <a href="https://github.com/openstack/marconi/blob/master/marconi/queues/storage/sqlalchemy/claims.py#L83" target="_blank">https://github.com/openstack/marconi/blob/master/marconi/queues/storage/sqlalchemy/claims.py#L83</a></div>
<div class="gmail_extra"> <br>
</div>
<div class="gmail_extra">Lastly, it looks like the Marconi storage drivers assume the storage back-end to be infinitely scalable. AFAICT, the mongo storage driver supports mongo's native sharding -- which I'm happy to see -- but the SQLA driver does not appear
to support anything equivalent for other back-ends, eg. MySQL. This relegates any deployment using the SQLA backend to the scale of "only what one database instance can handle". It's unsuitable for any large-scale deployment. Folks who don't want to use Mongo
are likely to use MySQL and will be promptly bitten by Marconi's lack of scalability with this back end. </div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">While there is a lot of room to improve the messaging around what/how/why, and I think a FAQ will be very helpful, I don't think that Marconi should graduate this cycle because:</div>
<div class="gmail_extra">(1) support for a non-AGPL-backend is a legal requirement [*] for Marconi's graduation;</div>
<div class="gmail_extra">
<div class="gmail_extra">(2) deploying Marconi with sqla+mysql will result in an incomplete and unscalable service.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>++</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">
<div class="gmail_extra">It's possible that I'm wrong about the scalability of Marconi with sqla + mysql. If anyone feels that this is going to perform blazingly fast on a single mysql db backend, please publish a benchmark and I'll be very happy to be proved
wrong. To be meaningful, it must have a high concurrency of clients creating and claiming messages with (num queues) << (num clients) << (num messages), and all clients polling on a reasonably short interval, based on what ever the recommended client-rate-limit
is. I'd like the test to be repeated with both Mongo and SQLA back-ends on the same hardware for comparison.<br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<div class="gmail_extra">Regards,</div>
<div class="gmail_extra">Devananda</div>
<div class="gmail_extra"><br>
</div>
<div class="gmail_extra">[*] <a href="https://wiki.openstack.org/wiki/Marconi/Incubation/Graduation#Legal_requirements" target="_blank">
https://wiki.openstack.org/wiki/Marconi/Incubation/Graduation#Legal_requirements</a></div>
</div>
<br>
</blockquote>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</span>
</body>
</html>