Open Stack

On 2017-10-18 12:15 PM, Yaguang Tang wrote:
> 
> We launched 300vms and each vm has about 10 metrics, OpenStack cluster 
> have 3 controllers and 2 compute nodes(ceph replication is set to 2).

seems smaller than my test, i have 20K metrics in my test

> 
> what we want to archive is to make all metric measures data get 
> processed as soon as possible, metric processing delay is set to 10s, 
> and ceilometer polling interval is 30s.

are you batching the data you push to gnocchi? in gnocchi4.1, the redis 
driver will (attempt to) process immediately, rather than cyclically 
using the metric processing delay.

> 
> when the backend of incoming and storeage is set to ceph, the average of 
> "gnocchi status"
> shows that there are around 7000 measures waiting to be process, but 
> when changing incoming and storage backend to Redis, the result of 
> gnocchi status shows unprocessed measures is around 200.

i should clarify, having a high gnocchi status is not a bad thing, ie, 
if you just send a large spike of measures, it's expected to jump. it's 
bad if never shrinks.

that said, how many workers do you have? i have 18 workers for 20K 
metrics and it takes 2minutes i believe? do you have debug enable? how 
long does it take to process metric?

when i tested gnocchi+ceph vs gnocchi+redis, i didn't see a very large 
difference in performance (redis was definitely better though). maybe 
it's your ceph environment?

> 
> we try to add more metricd process on every controller nodes, to improve 
> the performance of
> calculate and writing speed to storage backend but  have little effect.

performance should increase (relatively) proportionally. ie. if you 2x 
metricd, you should perform (almost) 2x quicker. if you add 5% more 
metricd, you should perform (almost) 5% quicker.

cheers,

-- 
gord