Open Stack

On 03/18/2014 08:09 AM, Nadya Privalova wrote:
> Hi folks,
> 
> I'd like to discuss Ceilometer's tempest situation with you.
> Now we have several patch sets on review that test core functionality of
> Ceilometer: notificaton and pollstering (topic
> https://review.openstack.org/#/q/status:open+project:openstack/tempest+branch:master+topic:bp/add-basic-ceilometer-tests,n,z).
> But there is a problem: Ceilometer performance is very poor on mysql and
> postgresql because of the bug
> https://bugs.launchpad.net/ceilometer/+bug/1291054. Mongo behaves much
> better even in single thread and I hope that it's performance will be
> enough to successfully run Ceilometer tempest tests.
> Let me explain in several words why tempest tests is mostly performance
> tests for Ceilometer. The thing is that Ceilometer service is running
> during all other nova, cinder and so on tests run. All the tests create
> instances, volumes and each creation produces a lot of notifications.
> Each notification is the entry to database. So Ceilometer cannot process
> such a big amount of notifications quickly. Ceilometer tests have
> 'telemetry' prefix and it means that they will be started in the last
> turn. And it makes situation even worst.
> So my proposal:
> 1. create a non-voting job with Mongo-backend
> 2. make sure that tests pass on Mongo
> 3. merge tests to tempest but skip that on postgres and mysql till
> bug/1291054 is resolved
> 4. make the new job 'voting'
> 
> The problem is only in Mongo installation. I have a cr
> https://review.openstack.org/#/c/81001/ that will allow us to install
> Mongo from deb. From the other hand there is
> https://review.openstack.org/#/c/74889/ that enables UCA. I'm
> collaborating with infra-team to make the decision ASAP because AFAIU we
> need tempest tests in Icehouse (for more discussion you are welcome to
> thread  [openstack-dev] Updating libvirt in gate jobs).
> 
> If you have any thoughts on this please share.

There is a fundamental problem here that the Ceilometer team requires a
version of Mongo that's not provided by the distro. We've taken a pretty
hard line on not requiring newer versions of non python stuff than the
distros we support actually have.

And the SQL backend is basically unusable from what I can tell.

So I'm -2 on injecting an arbitrary upstream Mongo in devstack.

What is preventing Ceilometer from bringing back support for the mongo
that you can get from 12.04? That seems like it should be the much
higher priority item. Then we could actually be gating Ceilometer
features on what the platforms can actually support. Then I'd be happy
to support a Mongo job running in tests.

Once that was done, we can start unpacking some of the other issues.

I'm not sure how changing to using 4 cores in the gate is going to
reduce the list command from 120s to 2s, so that doesn't really seem to
be the core issue (and is likely to just cause db deadlocks).

As long as Ceilometer says it supports SQL backends, it needs to do so
in a sane way. So that should still be gating.

	-Sean

-- 
Sean Dague
Samsung Research America
sean at dague.net / sean.dague at samsung.com
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140318/caca362c/attachment.pgp>