Open Stack

Hi Gordon，

Thanks for your test results， we investigate more on our env, finally it
turns out that our ceph cluster isn't work as expected.
which made gnocchi performance decrease a lot.

On Thu, Oct 19, 2017 at 1:09 AM, gordon chung <gord at live.ca> wrote:

>
>
> On 2017-10-18 12:15 PM, Yaguang Tang wrote:
> >
> > We launched 300vms and each vm has about 10 metrics, OpenStack cluster
> > have 3 controllers and 2 compute nodes(ceph replication is set to 2).
>
> seems smaller than my test, i have 20K metrics in my test
>
> >
> > what we want to archive is to make all metric measures data get
> > processed as soon as possible, metric processing delay is set to 10s,
> > and ceilometer polling interval is 30s.
>
> are you batching the data you push to gnocchi? in gnocchi4.1, the redis
> driver will (attempt to) process immediately, rather than cyclically
> using the metric processing delay.
>
> >
> > when the backend of incoming and storeage is set to ceph, the average of
> > "gnocchi status"
> > shows that there are around 7000 measures waiting to be process, but
> > when changing incoming and storage backend to Redis, the result of
> > gnocchi status shows unprocessed measures is around 200.
>
> i should clarify, having a high gnocchi status is not a bad thing, ie,
> if you just send a large spike of measures, it's expected to jump. it's
> bad if never shrinks.
>
> that said, how many workers do you have? i have 18 workers for 20K
> metrics and it takes 2minutes i believe? do you have debug enable? how
> long does it take to process metric?
>
> when i tested gnocchi+ceph vs gnocchi+redis, i didn't see a very large
> difference in performance (redis was definitely better though). maybe
> it's your ceph environment?
>
> >
> > we try to add more metricd process on every controller nodes, to improve
> > the performance of
> > calculate and writing speed to storage backend but  have little effect.
>
> performance should increase (relatively) proportionally. ie. if you 2x
> metricd, you should perform (almost) 2x quicker. if you add 5% more
> metricd, you should perform (almost) 5% quicker.
>
> cheers,
>
> --
> gord
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Tang Yaguang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171024/75d8994c/attachment.html>