[telemetry] Event Pipeline Config and Use Cases
Hello Openstack Telemetry Team, I'm currently working on re-enabling the Openstack Telemetry services for the StarlingX Openstack (STX-O) project. Openstack telemetry was integrated into STX-O long time ago, but it has been disabled for the past few years. Since the last time telemetry was used by STX-O, it seems that important updates happened, including: * Ceilometer API deprecation * Ceilometer Collector deprecation * Panko (event storage) deprecation My questions are mainly related to the last one (Panko deprecation): * Since Panko is deprecated and there is no actual replacement for event storage [1], is there any other possible telemetry use case to justify keeping the event pipeline enabled and generating data? * Are there any aodh alarming mechanisms that relies on events generated by Ceilometer event pipelines? If so, which "sinks.publishers" should be used to ensure the integration between the Ceilometer event pipeline and the aodh alarm service? * If event pipeline is enabled, what are the publishers currently supported for events? * The current event pipeline configuration (event_pipeline.yaml) in STX-O defines the "sinks.publishers" as "gnocchi://". However, based on Ceilometer doc, it looks like Gnocchi only supports samples/meters, it doesn't support events. Is my interpretation of the documentation correct or does Gnocchi support events and can be really used as a publisher for the event pipeline? [1] OpenInfra summit Vancouver 23 - Openstack Observability revamped<https://youtu.be/AhQJEBR5BUg?si=JbBuLG2iaaDZW7Iu&t=1590> Thank you in advance for any possible help! I'm looking forward to getting Openstack telemetry up and running on StarlingX Openstack. Best regards, Alex Figueirêdo.
Hi there, The ceilometer api has been deprecated if I remember for the kilo release. Ceilometer has been split into 4 components: - aodh for alarming - ceilometer for collecting and publishing metrics and events - gnocchi as metrics store - panko as events store. Gnocchi has been moved to independence (I could not find the corresponding blog post for more background). Moving it out of OpenStack may be seen as unfortunate, since it did not attract more developers but also created issues with testing. Panko did not scale, it was very slow. If you do not specify a receiver in the events pipeline, events will not be forwarded. What you could (and probably should) do if you rely on events is to create/update an events publisher that can publish to an http endpoint to store events e.g in Loki [2] and use its alarming mechanism. There are a couple of caveats though. If I had enough time, I would create an evaluator for aodh, similar to the prometheus evaluator[3] for alarming based on events. There may be a smarter way and would need a bit of investigation. You could also specify gnocchi as events store; that being said, it would create metrics with only one sample; not certain how smart that is. Hope that helps, Matthias [2] https://grafana.com/docs/loki/latest/ [3] https://github.com/openstack/aodh/commit/f932265290a4e923eac6111eb28578489c7...
Am 07.11.2024 um 15:43 schrieb Fernandes Figueiredo, Alex <Alex.FernandesFigueiredo@windriver.com>:
Hello Openstack Telemetry Team,
I'm currently working on re-enabling the Openstack Telemetry services for the StarlingX Openstack (STX-O) project. Openstack telemetry was integrated into STX-O long time ago, but it has been disabled for the past few years. Since the last time telemetry was used by STX-O, it seems that important updates happened, including: • Ceilometer API deprecation • Ceilometer Collector deprecation • Panko (event storage) deprecation My questions are mainly related to the last one (Panko deprecation): • Since Panko is deprecated and there is no actual replacement for event storage [1], is there any other possible telemetry use case to justify keeping the event pipeline enabled and generating data? • Are there any aodh alarming mechanisms that relies on events generated by Ceilometer event pipelines? If so, which "sinks.publishers" should be used to ensure the integration between the Ceilometer event pipeline and the aodh alarm service? • If event pipeline is enabled, what are the publishers currently supported for events? • The current event pipeline configuration (event_pipeline.yaml) in STX-O defines the "sinks.publishers" as "gnocchi://". However, based on Ceilometer doc, it looks like Gnocchi only supports samples/meters, it doesn't support events. Is my interpretation of the documentation correct or does Gnocchi support events and can be really used as a publisher for the event pipeline? [1] OpenInfra summit Vancouver 23 - Openstack Observability revamped
Thank you in advance for any possible help!
I'm looking forward to getting Openstack telemetry up and running on StarlingX Openstack.
Best regards, Alex Figueirêdo.
On 2024-11-09 18:20:08 +0100 (+0100), Matthias Runge wrote: [...]
Gnocchi has been moved to independence (I could not find the corresponding blog post for more background). Moving it out of OpenStack may be seen as unfortunate, since it did not attract more developers but also created issues with testing. [...]
I vaguely remember the blog post you're referring to, but couldn't manage to track it down either. The only public record I could turn up for the explanation at the time, however brief, is in the commit message of https://review.opendev.org/447438 which removed its repositories from the Telemetry team in March 2017:
After a lot of talk within the Gnocchi team, it appeared to us that Gnocchi, which has been wisely tagged as 'independent' since its inception, has a lot of potential usage outside of OpenStack directly.
Being part of the big tent helped the project to be built, but it now appears that it restrains its adoption and contribution from users outside of the OpenStack realm.
Therefore this patch un-lists Gnocchi from the OpenStack Telemetry project.
-- Jeremy Stanley
Hi Matthias, Thank you so much for taking your time to answer my questions! Your comments have made things clearer to me. About this:
I would create an evaluator for aodh, similar to the prometheus evaluator[3] for alarming based on events.
Taking a deeper look into the Aodh documentation, I found that Aodh already supports Alarms based on Ceilometer events [1] following the spec [2]. Is this similar to the Event Alarm evaluator you were talking about? If so, fortunately, it seems already implemented [3]. [1] https://docs.openstack.org/aodh/latest/contributor/event-alarm.html [2] https://review.opendev.org/c/openstack/telemetry-specs/+/172893 [3] https://blueprints.launchpad.net/ceilometer/+spec/event-alarm-evaluator Best regards, Alex Figueirêdo. ________________________________ From: Matthias Runge <mrunge@matthias-runge.de> Sent: Saturday, November 9, 2024 2:20 PM To: Fernandes Figueiredo, Alex <Alex.FernandesFigueiredo@windriver.com> Cc: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org>; Cervi, Thales Elero <ThalesElero.Cervi@windriver.com>; Waines, Greg <Greg.Waines@windriver.com> Subject: Re: [telemetry] Event Pipeline Config and Use Cases CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe. Hi there, The ceilometer api has been deprecated if I remember for the kilo release. Ceilometer has been split into 4 components: - aodh for alarming - ceilometer for collecting and publishing metrics and events - gnocchi as metrics store - panko as events store. Gnocchi has been moved to independence (I could not find the corresponding blog post for more background). Moving it out of OpenStack may be seen as unfortunate, since it did not attract more developers but also created issues with testing. Panko did not scale, it was very slow. If you do not specify a receiver in the events pipeline, events will not be forwarded. What you could (and probably should) do if you rely on events is to create/update an events publisher that can publish to an http endpoint to store events e.g in Loki [2] and use its alarming mechanism. There are a couple of caveats though. If I had enough time, I would create an evaluator for aodh, similar to the prometheus evaluator[3] for alarming based on events. There may be a smarter way and would need a bit of investigation. You could also specify gnocchi as events store; that being said, it would create metrics with only one sample; not certain how smart that is. Hope that helps, Matthias [2] https://grafana.com/docs/loki/latest/ [3] https://github.com/openstack/aodh/commit/f932265290a4e923eac6111eb28578489c7...
Am 07.11.2024 um 15:43 schrieb Fernandes Figueiredo, Alex <Alex.FernandesFigueiredo@windriver.com>:
Hello Openstack Telemetry Team,
I'm currently working on re-enabling the Openstack Telemetry services for the StarlingX Openstack (STX-O) project. Openstack telemetry was integrated into STX-O long time ago, but it has been disabled for the past few years. Since the last time telemetry was used by STX-O, it seems that important updates happened, including: • Ceilometer API deprecation • Ceilometer Collector deprecation • Panko (event storage) deprecation My questions are mainly related to the last one (Panko deprecation): • Since Panko is deprecated and there is no actual replacement for event storage [1], is there any other possible telemetry use case to justify keeping the event pipeline enabled and generating data? • Are there any aodh alarming mechanisms that relies on events generated by Ceilometer event pipelines? If so, which "sinks.publishers" should be used to ensure the integration between the Ceilometer event pipeline and the aodh alarm service? • If event pipeline is enabled, what are the publishers currently supported for events? • The current event pipeline configuration (event_pipeline.yaml) in STX-O defines the "sinks.publishers" as "gnocchi://". However, based on Ceilometer doc, it looks like Gnocchi only supports samples/meters, it doesn't support events. Is my interpretation of the documentation correct or does Gnocchi support events and can be really used as a publisher for the event pipeline? [1] OpenInfra summit Vancouver 23 - Openstack Observability revamped
Thank you in advance for any possible help!
I'm looking forward to getting Openstack telemetry up and running on StarlingX Openstack.
Best regards, Alex Figueirêdo.
Hi, You are right, after some digging, I found that it is implemented[4], though I haven't seen it working, but that may be my observation. Feel free to try it out and to report back. Matthias [4] https://review.opendev.org/q/topic:%22bp/event-alarm-evaluator%22 On 11/11/2024 14:05, Fernandes Figueiredo, Alex wrote:
Hi Matthias,
Thank you so much for taking your time to answer my questions! Your comments have made things clearer to me.
About this:
/> //I would create an evaluator for aodh, similar to the prometheus evaluator[3] for alarming based on events./
Taking a deeper look into the Aodh documentation, I found that Aodh already supports Alarms based on Ceilometer events [1] following the spec [2]. Is this similar to the Event Alarm evaluator you were talking about? If so, fortunately, it seems already implemented [3].
[1] https://docs.openstack.org/aodh/latest/contributor/event-alarm.html <https://docs.openstack.org/aodh/latest/contributor/event-alarm.html> [2] https://review.opendev.org/c/openstack/telemetry-specs/+/172893 <https://review.opendev.org/c/openstack/telemetry-specs/+/172893> [3] https://blueprints.launchpad.net/ceilometer/+spec/event-alarm- evaluator <https://blueprints.launchpad.net/ceilometer/+spec/event- alarm-evaluator>
Best regards, Alex Figueirêdo.
------------------------------------------------------------------------ *From:* Matthias Runge <mrunge@matthias-runge.de> *Sent:* Saturday, November 9, 2024 2:20 PM *To:* Fernandes Figueiredo, Alex <Alex.FernandesFigueiredo@windriver.com> *Cc:* openstack-discuss@lists.openstack.org <openstack- discuss@lists.openstack.org>; Cervi, Thales Elero <ThalesElero.Cervi@windriver.com>; Waines, Greg <Greg.Waines@windriver.com> *Subject:* Re: [telemetry] Event Pipeline Config and Use Cases CAUTION: This email comes from a non Wind River email account! Do not click links or open attachments unless you recognize the sender and know the content is safe.
Hi there,
The ceilometer api has been deprecated if I remember for the kilo release. Ceilometer has been split into 4 components: - aodh for alarming - ceilometer for collecting and publishing metrics and events - gnocchi as metrics store - panko as events store.
Gnocchi has been moved to independence (I could not find the corresponding blog post for more background). Moving it out of OpenStack may be seen as unfortunate, since it did not attract more developers but also created issues with testing. Panko did not scale, it was very slow.
If you do not specify a receiver in the events pipeline, events will not be forwarded. What you could (and probably should) do if you rely on events is to create/update an events publisher that can publish to an http endpoint to store events e.g in Loki [2] and use its alarming mechanism. There are a couple of caveats though. If I had enough time, I would create an evaluator for aodh, similar to the prometheus evaluator[3] for alarming based on events. There may be a smarter way and would need a bit of investigation. You could also specify gnocchi as events store; that being said, it would create metrics with only one sample; not certain how smart that is.
Hope that helps, Matthias
[2] https://grafana.com/docs/loki/latest/ <https://grafana.com/docs/ loki/latest/> [3] https://github.com/openstack/aodh/commit/ f932265290a4e923eac6111eb28578489c7dce33 <https://github.com/openstack/ aodh/commit/f932265290a4e923eac6111eb28578489c7dce33>
Am 07.11.2024 um 15:43 schrieb Fernandes Figueiredo, Alex <Alex.FernandesFigueiredo@windriver.com>:
Hello Openstack Telemetry Team,
I'm currently working on re-enabling the Openstack Telemetry services for the StarlingX Openstack (STX-O) project. Openstack telemetry was integrated into STX-O long time ago, but it has been disabled for the past few years. Since the last time telemetry was used by STX-O, it seems that important updates happened, including: • Ceilometer API deprecation • Ceilometer Collector deprecation • Panko (event storage) deprecation My questions are mainly related to the last one (Panko deprecation): • Since Panko is deprecated and there is no actual replacement for event storage [1], is there any other possible telemetry use case to justify keeping the event pipeline enabled and generating data? • Are there any aodh alarming mechanisms that relies on events generated by Ceilometer event pipelines? If so, which "sinks.publishers" should be used to ensure the integration between the Ceilometer event pipeline and the aodh alarm service? • If event pipeline is enabled, what are the publishers currently supported for events? • The current event pipeline configuration (event_pipeline.yaml) in STX-O defines the "sinks.publishers" as "gnocchi://". However, based on Ceilometer doc, it looks like Gnocchi only supports samples/meters, it doesn't support events. Is my interpretation of the documentation correct or does Gnocchi support events and can be really used as a publisher for the event pipeline? [1] OpenInfra summit Vancouver 23 - Openstack Observability revamped
Thank you in advance for any possible help!
I'm looking forward to getting Openstack telemetry up and running on StarlingX Openstack.
Best regards, Alex Figueirêdo.
-- Matthias Runge <mrunge@matthias-runge.de>
participants (3)
-
Fernandes Figueiredo, Alex
-
Jeremy Stanley
-
Matthias Runge