<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Mar 20, 2014 at 3:03 PM, Alexei Kornienko <span dir="ltr"><<a href="mailto:alexei.kornienko@gmail.com" target="_blank">alexei.kornienko@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hello,<br>
<br>
We've done some profiling and results are quite interesting:<br>
during 1,5 hour ceilometer inserted 59755 events (59755 calls to
record_metering_data)<br>
this calls resulted in total 2591573 SQL queries.<br>
<br>
And the most interesting part is that 291569 queries were ROLLBACK
queries.<br>
We do around 5 rollbacks to record a single event!<br>
<br>
I guess it means that MySQL backend is currently totally unusable
in production environment.<br></div></div></blockquote><div><br></div><div>It should be noticed that SQLAlchemy is horrible for performance, in nova we usually see sqlalchemy overheads of well over 10x (time nova.db.api call vs the time MySQL measures when slow log is recording everything).</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div bgcolor="#FFFFFF" text="#000000"><div>
<br>
Please find a full profiling graph attached.<br>
<br>
Regards,<div><div class="h5"><br>
<br>
On 03/20/2014 10:31 PM, Sean Dague wrote:<br>
</div></div></div>
<blockquote type="cite"><div><div class="h5">
<pre>On 03/20/2014 01:01 PM, David Kranz wrote:
</pre>
<blockquote type="cite">
<pre>On 03/20/2014 12:31 PM, Sean Dague wrote:
</pre>
<blockquote type="cite">
<pre>On 03/20/2014 11:35 AM, David Kranz wrote:
</pre>
<blockquote type="cite">
<pre>On 03/20/2014 06:15 AM, Sean Dague wrote:
</pre>
<blockquote type="cite">
<pre>On 03/20/2014 05:49 AM, Nadya Privalova wrote:
</pre>
<blockquote type="cite">
<pre>Hi all,
First of all, thanks for your suggestions!
To summarize the discussions here:
1. We are not going to install Mongo (because "is's wrong" ?)
</pre>
</blockquote>
<pre>We are not going to install Mongo "not from base distribution", because
we don't do that for things that aren't python. Our assumption is
dependent services come from the base OS.
That being said, being an integrated project means you have to be able
to function, sanely, on an sqla backend, as that will always be part of
your gate.
</pre>
</blockquote>
<pre>This is a claim I think needs a bit more scrutiny if by "sanely" you
mean "performant". It seems we have an integrated project that no one
would deploy using the sql db driver we have in the gate. Is any one
doing that? Is having a scalable sql back end a goal of ceilometer?
More generally, if there is functionality that is of great importance to
any cloud deployment (and we would not integrate it if we didn't think
it was) that cannot be deployed at scale using sqla, are we really going
to say it should not be a part of OpenStack because we refuse, for
whatever reason, to run it in our gate using a driver that would
actually be used? And if we do demand an sqla backend, how much time
should we spend trying to optimize it if no one will really use it?
Though the slow heat job is a little different because the slowness
comes directly from running real use cases, perhaps we should just set
up a "slow ceilometer" job if the sql version is too slow for its budget
in the main job.
It seems like there is a similar thread, at least in part, about this
around marconi.
</pre>
</blockquote>
<pre>We required a non mongo backend to graduate ceilometer. So I don't think
it's too much to ask that it actually works.
If the answer is that it will never work and it was a checkbox with no
intent to make it work, then it should be deprecated and removed from
the tree in Juno, with a big WARNING that you shouldn't ever use that
backend. Like Nova now does with all the virt drivers that aren't tested
upstream.
Shipping in tree code that you don't want people to use is bad for
users. Either commit to making it work, or deprecate it and remove it.
I don't see this as the same issue as the slow heat job. Heat,
architecturally, is going to be slow. It spins up real OSes and does
real thinks to them. There is no way that's ever going to be fast, and
the dedicated job was a recognition that to support this level of
services in OpenStack we need to give them more breathing room.
</pre>
</blockquote>
<pre>Peace. I specifically noted that difference in my original comment. And
for that reason the heat slow job may not be temporary.
</pre>
<blockquote type="cite">
<pre>Architecturally Ceilometer should not be this expensive. We've got some
data showing it to be aberrant from where we believe it should be. We
should fix that.
</pre>
</blockquote>
<pre>There are plenty of cases where we have had code that passes gate tests
with acceptable performance but falls over in real deployment. I'm just
saying that having a driver that works ok in the gate but does not work
for real deployments is of no more value that not having it at all.
Maybe less value.
How do you propose to solve the problem of getting more ceilometer tests
into the gate in the short-run? As a practical measure l don't see why
it is so bad to have a separate job until the complex issue of whether
it is possible to have a real-world performant sqla backend is resolved.
Or did I miss something and it has already been determined that sqla
could be used for large-scale deployments if we just fixed our code?
</pre>
</blockquote>
<pre>I think right now the ball is back in the ceilometer court to do some
performance profiling, and lets see what comes of that. I don't think
we're getting more test before the release in any real way.
</pre>
<blockquote type="cite">
<blockquote type="cite">
<pre>Once we get a base OS in the gate that lets us direct install mongo from
base packages, we can also do that. Or someone can 3rd party it today.
Then we'll even have comparative results to understand the differences.
</pre>
</blockquote>
<pre>Yes. Do you know which base OS's are candidates for that?
</pre>
</blockquote>
<pre>Ubuntu 14.04 will have a sufficient level of Mongo, so some time in the
Juno cycle we should have it in the gate.
-Sean
</pre>
<br>
<fieldset></fieldset>
<br>
</div></div><div class=""><pre>_______________________________________________
OpenStack-dev mailing list
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</div></blockquote>
<br>
</div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>