Open Stack

On 04/03/2016 5:15 PM, Stig Telfer wrote:
>
>> On 4 Mar 2016, at 15:40, gordon chung <gord at live.ca> wrote:
>>
>>> One part of the documentation set that we were missing was a guide to how to migrate from ceilometer to a ceilometer/gnocchi combination (which I understand is the ultimate architecture). We would like to migrate the historical data we have stored in ceilometer.
>>>
>>> The main line documentation (such as http://docs.openstack.org/liberty/config-reference/content/) does not yet contain details on the Gnocchi configuration so some people may miss this option when following the docs. The gnocchi.xyz has good content but it is not the standard configuration guides. I guess once Gnocchi is into the Big Tent then this sort of integration can occur.
>>>
>>> It’s early days yet so we don’t yet have the feeling for running Gnocchi at scale.
>>>
>>> Tim
>>
>> hi,
>>
>> there isn't an active migration path between Ceilometer's legacy
>> database to Gnocchi. it was discussed but given the limited resources we
>> have (contributors are welcomed) we just do not have the bandwidth
>> currently to build a tool to help port data over.
>>
>> Gnocchi is not a 1:1 replacement of Ceilometer's old API, it does not
>> capture full-fidelity of data and requires defined archive policy rules,
>> which makes a migration tool necessary. it is mainly intended to replace
>> the 'ceilometer statistics' use case of the old API.
>>
>> we were hoping to get a reference architecture this cycle but were
>> delayed due to resources being pulled to other tasks. currently, the
>> best option is to take a look at the devstack plugins for Gnocchi[1] and
>> Ceilometer[2] (requires bash knowledge) to get an idea of how they're
>> set up to work together in the gate.
>>
>> as with all new things, it's recommended you test this out first, to
>> learn new API[3] and find potential bugs. you can configure the
>> Ceilometer collector to write to both existing db and Gnocchi so your
>> current usage is not interrupted.
>>
>> [1] https://github.com/openstack/gnocchi/blob/master/devstack/plugin.sh
>> [2] https://github.com/openstack/ceilometer/blob/master/devstack/plugin.sh
>> [3] http://gnocchi.xyz/rest.html
>>
>> cheers,
>> --
>> gord
>
> We evaluated Monasca as an alternative for monitoring.  One of its appeals is that through its use of Kafka we get logging, events and time-series telemetry data streams all in one place (we send syslog through the same Kafka instance).  The attraction of this was that it would then be feasible to create a system for multi-variate alert triggering and anomaly detection, using a single API to access near-real-time data from arbitrary and diverse sources from across the data centre.
>
> Not that that system ever got created, ahem, but through bringing the data sources together I think it was brought a step closer…
>
> Gord, if Gnocchi is functionality split off from Ceilometer, would using Gnocchi require API changes to any client subscribing to sources of both monitoring and event data?
>
> Huge thanks to you Anita for starting this discussion.
>

hi,

sorry, i'm not familiar with the internal design decisions made by 
another company so my answers might be quite vague. i do know there is a 
community effort to actually support Kafka[1] if you have a requirement 
for it. this actually makes a lot more sense as it doesn't matter if 
your system supports higher throughput downstream if upstream is your 
bottleneck.

Gnocchi is designed to store metric data (not events). Ceilometer 
provides the data gathering/normalisation functionality. if you are 
using another project to handle storage, you should not need to worry 
about Gnocchi's API or Ceilometer's existing API; you should really just 
be publishing the data straight from Ceilometer to your own storage. 
that way your storage and API is optimised to work together and match 
your own requirements. if you are using Ceilometer as storage in 
addition to other options, then yes, there are distinct API differences 
between Ceilometer and Gnocchi. from your description, it seems like you 
are just subscribing to the data so you should not require Gnocchi or 
Ceilometer API and should be following 'external systems' path[2].

[1] 
https://specs.openstack.org/openstack/oslo-specs/specs/liberty/adding-kafka-support.html
[2] 
http://docs.openstack.org/developer/ceilometer/architecture.html#high-level-architecture

cheers,

-- 
gord