On 16/04/2025 21:04, Dmitriy Rabotyagov wrote:

Hey,

Have a comment on one AI from the list.

...

AI: (jgilaber) Mark Monasca and Grafana as deprecated, unless someone steps up to maintain them, which should include a minimal CI job running.

So eventually, on OpenStack-Ansible we were planning to revive the Watcher role support to the project. How we usually test deployment, is by spawning an all-in-one environment with drivers and executing a couple of tempest scenarios to ensure basic functionality of the service.

With that, having a native OpenStack telemetry datastore is very beneficial for such goal, as we already do maintain means for spawning telemetry stack. While a requirement for Prometheus will be unfortunate for us at least.

While I was writing that, I partially realized that testing Watcher on all-in-one is pretty much impossible as well...

you can certenly test some fo watcher with an all in one deployment i.e. the apis and you can use the dummy test stragies. but ya in general like nova you need at least 2 nodes to be able to test it properly ideally 3 so that if your doing a live migration there is actully a choice of host. in general however watcher like heat just asks nova to actully move the vms. sure it will ask nova to move it to a specific host but fundementaly if you have tested live migration with nova via tempest seperatly there is no reason to expcect it would not work for live migratoin tirggred by watcher or heat or anything else that jsut calls novas api. so you could still get some valual testing in an all in one but ideally there woudl be at least 2 comptue hosts.

But at the very least, I can propose looking into adding an OSA job with Gnocchi as NV to the project, to show the state of the deployment with this driver.

well gnocchi is also not a native OpenStack telemetry datastore, it left our community to pursue its own goals and is now a third party datastore just like Grafana or Prometheus. monasca is currently marked as inactive https://review.opendev.org/c/openstack/governance/+/897520 and is in the process of being retired. but it also has no testing on the watcher side to the combination of the two is why we are deprecating it going forward. if both change im happy to see the support continue. Gnocchi has testing but we are not actively working on extending its functionality going forward. as long as it continues to work i see no reason to change its support status. watcher has quite a lot of untested integrations which is unfortunate we are planning to build out a feature/test/support matrix in the docs this cycle but for example watcher can integrate with both ironic an canonical maas component to do some level of host power management. none of that is tested and we are likely going to mark them as experimental and reflect on if we can continue to support them or not going forward. it also has the ability to do cinder storage pool balancing which is i think also untested write now. one of the things we hope to do is extend the exsitign testing in our current jobs to cover gaps like that where it is practical to do so. but creating a devstack plugin to deploy maas with fake infrastructure is likely alot more then we can do with our existing contributors so expect that to go to experimental then deprecated and finally it will be removed if no one turns up to support it. ironic is in the same boat however there are devstack jobs with fake ironic nodes so i could see a path to use having an ironic job down the line. its just not high on our current priority list to adress the support status or testing of this currently. eventlet removal and other techdebt/community goals are defintly higher but i hop the new supprot/testing matrix will at least help folks make informed descions or what feature to use and what backend are recommended going forward.

On Wed, 16 Apr 2025, 21:53 Douglas Viroel, <viroel@gmail.com> wrote:

Hello everyone,

Last week's PTG had very interesting topics. Thank you all that joined. The Watcher PTG etherpad with all notes is available here: https://etherpad.opendev.org/p/apr2025-ptg-watcher Here is a summary of the discussions that we had, including the great cross-project sessions with Telemetry, Horizon and Nova team:

Tech Debt (chandankumar/sean-k-mooney) ================================= a) Croniter

* Project is being abandoned as per https://pypi.org/project/croniter/#disclaimer * Watcher uses croniter to calculate a new schedule time to run an audit (continuous). It is also used to validate cron like syntax * Agreed: replace croniter with appscheduler's cron methods. * *AI*: (chandankumar) Fix in master branch and backport to 2025.1

b) Support status of Watcher Datasources

* Only Gnocchi and Prometheus have CI job running tempest tests (with scenario tests) * Monaska is inactive since 2024.1 * *AI*: (jgilaber) Mark Monasca and Grafana as deprecated, unless someone steps up to maintain them, which should include a minimal CI job running. * *AI*: (dviroel) Document a support matrix between Strategies and Datasources, which ones are production ready or experimental, and testing coverage.

c) Eventlet Removal

* Team is going to look at how the eventlet is used in Watcher and start a PoC of its removal. * Chandan Kumar and dviroel volunteer to help in this effort. * Planned for 2026.1 cycle.

Workflow/API Improvements (amoralej) ============================== a) Actions states

* Currently Actions updates from Pending to Succeeded or Failed, but these do not cover some important scenarios * If an Action's pre_conditions fails, the action is set to FAILED, but for some scenarios, it could be just SKIPPED and continue the workflow. * Proposal: New SKIPPED state for action. E.g: In a Nova Migration Action, if the instance doesn't exist in the source host, it can be skipped instead of fail. * Proposal: Users could also manually skip specific actions from an action plan. * A skip_reason field could also be added to document the reason behind the skip: user's request, pre-condition check, etc. * *AI*: (amoralej) Create a spec to describe the proposed changes.

b) Meaning of SUCCEEDED state in Action Plan

* Currently means that all actions are triggered, even if all of them fail, which can be confusing for users. * Docs mention that SUCCEEDED state means that all actions have been successfully executed. * *AI*: (amoralej) Document the current behavior as a bug (Priority High) o done: https://bugs.launchpad.net/watcher/+bug/2106407

Watcher-Dashboard: Priorities to next release (amoralej) =========================================== a) Add integration/functional tests

* Project is missing integration/functional tests and a CI job running against changes in the repo * No general conclusion and we will follow up with Horizon team * *AI*: (chandankumar/rlandy) sync with Horizon team about testing the plugin with horizon. * *AI*: (chandankumar/rlandy) devstack job running on new changes for watcher-dashboard repo.

b) Add parameters to Audits

* It is missing on the watcher-dashboard side. Without it, it is not possible to define some important parameters. * Should be addressed by a blueprint * Contributors to this feature: chandankumar

Watcher cluster model collector improvement ideas (dviroel) =============================================

* Brainstorm ideas to improve watcher collector process, since we still see a lot of issues due to outdated models when running audits * Both scheduled model update and event-based updates are enabled in CI today * It is unknown the current state of event-based updates from Nova notification. Code needs to be reviewed and improvements/fixes can be proposed o e.g: https://bugs.launchpad.net/watcher/+bug/2104220/comments/3 - We need to check if we are processing the right notifications of if is a bug on Nova * Proposal: Refresh the model before running an audit. A rate limit should be considered to avoid too many refreshments. * *AI*: (dviroel) new spec for cluster model refresh, based on audit trigger * *AI:* (dviroel) investigate the processing of nova events in Watcher

Watcher and Nova's visible constraints (dviroel) ====================================

* Currently, Watcher can propose solutions that include server migrations that violate some Nova constraints like: scheduler_hints, server_groups, pinned_az, etc. * In Epoxy release, Nova's API was improved to also show scheduler_hints and image_properties, allowing external services, like watcher, to query and use this information when calculating new solutions. o https://docs.openstack.org/releasenotes/nova/2025.1.html#new-features * Proposal: Extend compute instance model to include new properties, which can be retrieved via novaclient. Update strategies to filter invalid migration destinations based on these new properties. * *AI*: (dviroel) Propose a spec to better document the proposal. No API changes are expected here.

Replacement for noisy neighbor policy (jgilaber) ====================================

* The existing noisy neighbor strategy is based on L3 Cache metrics, which is not available anymore, since the support for it was dropped from the kernel and from Nova. * In order to keep this strategy, new metrics need to be considered: cpu_steal? io_wait? cache_misses? * *AI*: (jgilaber) Mark the strategy as deprecated during this cycle * *AI*: (TBD) Identify new metrics to be used * *AI*: (TBD) Work on a replacement for the current strategy

Host Maintenance strategy new use case (jeno8) =====================================

* New use case for Host Maintenance strategy: instance with ephemeral disks should not be migrated at all. * Spec proposed: https://review.opendev.org/c/openstack/watcher-specs/+/943873 o New action to stop instances when both live/cold migration are disabled by the user * *AI*: (All) Review the spec and continue with discussion there.

Missing Contributor Docs (sean-k-mooney) ================================

* Doc missing: Scope of the project, e.g: https://docs.openstack.org/nova/latest/contributor/project-scope.html * *AI*: (rlandy) Create a scope of the project doc for Watcher * Doc missing: PTL Guide, e.g: https://docs.openstack.org/nova/latest/contributor/ptl-guide.html * *AI*: (TBD) Create a PTL Guide for Watcher project * Document: When to create a spec vs blueprint vs bug * *AI*: (TBD) Create a doc section to describe the process based on what is being modified in the code.

Retrospective ==========

* The DPL approach seems to be working for Watcher * New core members added: sean-k-mooney, dviroel, marios and chandankumar o We plan to add more cores in the next cycle, based on reviews and engagement. o We plan to remove not active members in the 2 last cycles (starting at 2026.1) * A new datasource was added: Prometheus * Prometheus job now also runs scenario tests, along with Gnocchi. * We triaged all old bugs from launchpad * Needs improvement: o current team is still learning about details in the code, much of the historical knowledge was lost with the previous maintainers o core team still needs to grow o we need to focus on creating stable releases

Cross-project session with Horizon team ===============================

* Combined session with Telemetry and Horizon team, focused on how to provide a tenant and an admin dashboard to visualize metrics. * Watcher team presented some ideas of new panels for both admin and tenants, and sean-k-mooney raised a discussion about frameworks that can be used to implement them * Use-cases that were discussed: o a) Admin would benefit from a visualization of the infrastructure utilization (real usage metrics), so they can identify bottlenecks and plan optimization o b) A tenant would like to view their workload performance, checking real usage of cpu/ram/disk of instances, to proper adjust their resources allocation. o c) An admin user of watcher service would like to visualize metrics generated by watcher strategies like standard deviation of host metrics. * sean-k-mooney presented an initial PoC on how a Hypervisor Metrics dashboard would look like. * Proposal for next steps: o start a new horizon plugin as an official deliverable of telemetry project o still unclear which framework to use for building charts o dashboard will integrate with Prometheus, as metric store o it is expected that only short term metrics will be supported (7 days) o python-observability-client will be used to query Prometheus

Cross-project session with Nova team =============================

* sean-k-mooney led topics on how to evolve Nova to better assist other services, like Watcher, to take actions on instances. The team agreed on a proposal of using the existing metadata API to annotate instance's supported lifecycle operations. This information is very useful to improve Watcher's strategy's algorithms. Some example of instance's metadata could be: o lifecycle:cold-migratable=true|false o ha:maintenance-strategy:in_place|power_off|migrate * It was discussed that Nova could infer which operations are valid or not, based on information like: virt driver, flavor, image properties, etc. This feature was initially named 'instance capabilities' and will require a spec for further discussions. * Another topic of interest, also raised by Sean, was about adding new standard traits to resource providers, like PRESSURE_CPU and PRESSURE_DISK. These traits can be used to weight hosts when placing new VMs. Watcher and the libvirt driver could work on annotating them, but the team generally agreed that the libvirt driver is preferred here. * More info at Nova PTG etherpad [0] and sean's summary blog [1]

[0] https://etherpad.opendev.org/p/r.bf5f1185e201e31ed8c3adeb45e3cf6d [1] https://www.seanmooney.info/blog/2025.2-ptg/#watcher-topics

Please let me know if I missed something. Thanks!

-- Douglas Viroel - dviroel

On 17/04/2025 13:17, Dmitriy Rabotyagov wrote:

...

well gnocchi is also not a native OpenStack telemetry datastore, it left our community to pursue its own goals and is now a third party datastore just like Grafana or Prometheus. Yeah, well, true. Is still somehow treated as the "default" thing with Telemetry, likely due to existing integration with Keystone and multi-tenancy support. And beyond it - all other options become opinionated too fast - ie, some do OpenTelemetry, some do Zabbix, VictoriaMetrics, etc. As pretty much from what I got as well, is that still relies on Ceilometer metrics? And then Prometheus is obviously not the best storage for them, as it requires to have pushgatgeway, and afaik prometheus maintainers are strictly against "push" concept to it and treat it as conceptually wrong (on contrary to OpenTelemetry).

i dont know the detail but i know there is work planned for native supprot of Prometheus scrpe endpoint in ceilometer so while you currently need to use SG-core to provide that integration there is a plan to remove the need for sgcore going forward. https://etherpad.opendev.org/p/r.72ac6a7268e4b9d854f75715adede80c#L28 i dont see a spec proposed yet but there is an olde one form 2 years ago https://review.opendev.org/c/openstack/telemetry-specs/+/845485/4/specs/zed/... there is also a plan to provide keystone integration and mutli tenancy https://etherpad.opendev.org/p/r.72ac6a7268e4b9d854f75715adede80c#L84

So the metric timestamp issue is to remain unaddressed. So that's why I'd see leaving Gnocchi as "base" implementation might be valuable (and very handy for us, as we don't need to implement a prometheus job specifically for Watcher).

watcher, aodh, and cloud kitty i believe all  have some level if support for Prometheus but they can also use other backends. in not sure what level of enablement they have in osa.

...

but for example watcher can integrate with both ironic an canonical maas component to do some level of host power management. That sounds really interesting... We do maintain infrastructure using MAAS and playing with such integration will be extremely interesting. I hope I will be able to get some time for this though...

the current maas integration has 3 problems. 1 a lack of testing, 2 a lack of documentation and 3 it somehow managed to introduce asysnio in a project that uses eventlet in a release of eventlet that did not support asyncio so im very nervious that that is broken or will break in the future. this is the entrity of the support https://review.opendev.org/c/openstack/watcher/+/898790 there are no docs and no spec... so this should definitely be considered "experimental" at best today.

чт, 17 апр. 2025 г. в 13:52, Sean Mooney <smooney@redhat.com>:

...

On 16/04/2025 21:04, Dmitriy Rabotyagov wrote:

...

Hey,

Have a comment on one AI from the list.

...

AI: (jgilaber) Mark Monasca and Grafana as deprecated, unless someone steps up to maintain them, which should include a minimal CI job running.

So eventually, on OpenStack-Ansible we were planning to revive the Watcher role support to the project. How we usually test deployment, is by spawning an all-in-one environment with drivers and executing a couple of tempest scenarios to ensure basic functionality of the service.

With that, having a native OpenStack telemetry datastore is very beneficial for such goal, as we already do maintain means for spawning telemetry stack. While a requirement for Prometheus will be unfortunate for us at least.

While I was writing that, I partially realized that testing Watcher on all-in-one is pretty much impossible as well...

you can certenly test some fo watcher with an all in one deployment

i.e. the apis and you can use the dummy test stragies.

but ya in general like nova you need at least 2 nodes to be able to test it properly ideally 3

so that if your doing a live migration there is actully a choice of host.

in general however watcher like heat just asks nova to actully move the vms.

sure it will ask nova to move it to a specific host but fundementaly if you have

tested live migration with nova via tempest seperatly there is no reason to expcect

it would not work for live migratoin tirggred by watcher or heat or anything else that

jsut calls novas api.

so you could still get some valual testing in an all in one but ideally there woudl be at least 2 comptue hosts.

...

But at the very least, I can propose looking into adding an OSA job with Gnocchi as NV to the project, to show the state of the deployment with this driver.

well gnocchi is also not a native OpenStack telemetry datastore, it left our community to pursue its own goals and is now a third party datastore

just like Grafana or Prometheus.

monasca is currently marked as inactive https://review.opendev.org/c/openstack/governance/+/897520 and is in the process of being retired.

but it also has no testing on the watcher side to the combination of the two is why we are deprecating it going forward.

if both change im happy to see the support continue.

Gnocchi has testing but we are not actively working on extending its functionality going forward.

as long as it continues to work i see no reason to change its support status.

watcher has quite a lot of untested integrations which is unfortunate

we are planning to build out a feature/test/support matrix in the docs this cycle

but for example watcher can integrate with both ironic an canonical maas component

to do some level of host power management. none of that is tested and we are likely going

to mark them as experimental and reflect on if we can continue to support them or not going forward.

it also has the ability to do cinder storage pool balancing which is i think also untested write now.

one of the things we hope to do is extend the exsitign testing in our current jobs to cover gaps like

that where it is practical to do so. but creating a devstack plugin to deploy maas with fake infrastructure

is likely alot more then we can do with our existing contributors so expect that to go to experimental then

deprecated and finally it will be removed if no one turns up to support it.

ironic is in the same boat however there are devstack jobs with fake ironic nodes so i

could see a path to use having an ironic job down the line. its just not high on our current priority

list to adress the support status or testing of this currently.

eventlet removal and other techdebt/community goals are defintly higher but i hop the new supprot/testing

matrix will at least help folks make informed descions or what feature to use and what backend are

recommended going forward.

...

On Wed, 16 Apr 2025, 21:53 Douglas Viroel, <viroel@gmail.com> wrote:

Hello everyone,

Last week's PTG had very interesting topics. Thank you all that joined. The Watcher PTG etherpad with all notes is available here: https://etherpad.opendev.org/p/apr2025-ptg-watcher Here is a summary of the discussions that we had, including the great cross-project sessions with Telemetry, Horizon and Nova team:

Tech Debt (chandankumar/sean-k-mooney) ================================= a) Croniter

* Project is being abandoned as per https://pypi.org/project/croniter/#disclaimer * Watcher uses croniter to calculate a new schedule time to run an audit (continuous). It is also used to validate cron like syntax * Agreed: replace croniter with appscheduler's cron methods. * *AI*: (chandankumar) Fix in master branch and backport to 2025.1

b) Support status of Watcher Datasources

* Only Gnocchi and Prometheus have CI job running tempest tests (with scenario tests) * Monaska is inactive since 2024.1 * *AI*: (jgilaber) Mark Monasca and Grafana as deprecated, unless someone steps up to maintain them, which should include a minimal CI job running. * *AI*: (dviroel) Document a support matrix between Strategies and Datasources, which ones are production ready or experimental, and testing coverage.

c) Eventlet Removal

* Team is going to look at how the eventlet is used in Watcher and start a PoC of its removal. * Chandan Kumar and dviroel volunteer to help in this effort. * Planned for 2026.1 cycle.

Workflow/API Improvements (amoralej) ============================== a) Actions states

* Currently Actions updates from Pending to Succeeded or Failed, but these do not cover some important scenarios * If an Action's pre_conditions fails, the action is set to FAILED, but for some scenarios, it could be just SKIPPED and continue the workflow. * Proposal: New SKIPPED state for action. E.g: In a Nova Migration Action, if the instance doesn't exist in the source host, it can be skipped instead of fail. * Proposal: Users could also manually skip specific actions from an action plan. * A skip_reason field could also be added to document the reason behind the skip: user's request, pre-condition check, etc. * *AI*: (amoralej) Create a spec to describe the proposed changes.

b) Meaning of SUCCEEDED state in Action Plan

* Currently means that all actions are triggered, even if all of them fail, which can be confusing for users. * Docs mention that SUCCEEDED state means that all actions have been successfully executed. * *AI*: (amoralej) Document the current behavior as a bug (Priority High) o done: https://bugs.launchpad.net/watcher/+bug/2106407

Watcher-Dashboard: Priorities to next release (amoralej) =========================================== a) Add integration/functional tests

* Project is missing integration/functional tests and a CI job running against changes in the repo * No general conclusion and we will follow up with Horizon team * *AI*: (chandankumar/rlandy) sync with Horizon team about testing the plugin with horizon. * *AI*: (chandankumar/rlandy) devstack job running on new changes for watcher-dashboard repo.

b) Add parameters to Audits

* It is missing on the watcher-dashboard side. Without it, it is not possible to define some important parameters. * Should be addressed by a blueprint * Contributors to this feature: chandankumar

Watcher cluster model collector improvement ideas (dviroel) =============================================

* Brainstorm ideas to improve watcher collector process, since we still see a lot of issues due to outdated models when running audits * Both scheduled model update and event-based updates are enabled in CI today * It is unknown the current state of event-based updates from Nova notification. Code needs to be reviewed and improvements/fixes can be proposed o e.g: https://bugs.launchpad.net/watcher/+bug/2104220/comments/3 - We need to check if we are processing the right notifications of if is a bug on Nova * Proposal: Refresh the model before running an audit. A rate limit should be considered to avoid too many refreshments. * *AI*: (dviroel) new spec for cluster model refresh, based on audit trigger * *AI:* (dviroel) investigate the processing of nova events in Watcher

Watcher and Nova's visible constraints (dviroel) ====================================

* Currently, Watcher can propose solutions that include server migrations that violate some Nova constraints like: scheduler_hints, server_groups, pinned_az, etc. * In Epoxy release, Nova's API was improved to also show scheduler_hints and image_properties, allowing external services, like watcher, to query and use this information when calculating new solutions. o https://docs.openstack.org/releasenotes/nova/2025.1.html#new-features * Proposal: Extend compute instance model to include new properties, which can be retrieved via novaclient. Update strategies to filter invalid migration destinations based on these new properties. * *AI*: (dviroel) Propose a spec to better document the proposal. No API changes are expected here.

Replacement for noisy neighbor policy (jgilaber) ====================================

* The existing noisy neighbor strategy is based on L3 Cache metrics, which is not available anymore, since the support for it was dropped from the kernel and from Nova. * In order to keep this strategy, new metrics need to be considered: cpu_steal? io_wait? cache_misses? * *AI*: (jgilaber) Mark the strategy as deprecated during this cycle * *AI*: (TBD) Identify new metrics to be used * *AI*: (TBD) Work on a replacement for the current strategy

Host Maintenance strategy new use case (jeno8) =====================================

* New use case for Host Maintenance strategy: instance with ephemeral disks should not be migrated at all. * Spec proposed: https://review.opendev.org/c/openstack/watcher-specs/+/943873 o New action to stop instances when both live/cold migration are disabled by the user * *AI*: (All) Review the spec and continue with discussion there.

Missing Contributor Docs (sean-k-mooney) ================================

* Doc missing: Scope of the project, e.g: https://docs.openstack.org/nova/latest/contributor/project-scope.html * *AI*: (rlandy) Create a scope of the project doc for Watcher * Doc missing: PTL Guide, e.g: https://docs.openstack.org/nova/latest/contributor/ptl-guide.html * *AI*: (TBD) Create a PTL Guide for Watcher project * Document: When to create a spec vs blueprint vs bug * *AI*: (TBD) Create a doc section to describe the process based on what is being modified in the code.

Retrospective ==========

* The DPL approach seems to be working for Watcher * New core members added: sean-k-mooney, dviroel, marios and chandankumar o We plan to add more cores in the next cycle, based on reviews and engagement. o We plan to remove not active members in the 2 last cycles (starting at 2026.1) * A new datasource was added: Prometheus * Prometheus job now also runs scenario tests, along with Gnocchi. * We triaged all old bugs from launchpad * Needs improvement: o current team is still learning about details in the code, much of the historical knowledge was lost with the previous maintainers o core team still needs to grow o we need to focus on creating stable releases

Cross-project session with Horizon team ===============================

* Combined session with Telemetry and Horizon team, focused on how to provide a tenant and an admin dashboard to visualize metrics. * Watcher team presented some ideas of new panels for both admin and tenants, and sean-k-mooney raised a discussion about frameworks that can be used to implement them * Use-cases that were discussed: o a) Admin would benefit from a visualization of the infrastructure utilization (real usage metrics), so they can identify bottlenecks and plan optimization o b) A tenant would like to view their workload performance, checking real usage of cpu/ram/disk of instances, to proper adjust their resources allocation. o c) An admin user of watcher service would like to visualize metrics generated by watcher strategies like standard deviation of host metrics. * sean-k-mooney presented an initial PoC on how a Hypervisor Metrics dashboard would look like. * Proposal for next steps: o start a new horizon plugin as an official deliverable of telemetry project o still unclear which framework to use for building charts o dashboard will integrate with Prometheus, as metric store o it is expected that only short term metrics will be supported (7 days) o python-observability-client will be used to query Prometheus

Cross-project session with Nova team =============================

* sean-k-mooney led topics on how to evolve Nova to better assist other services, like Watcher, to take actions on instances. The team agreed on a proposal of using the existing metadata API to annotate instance's supported lifecycle operations. This information is very useful to improve Watcher's strategy's algorithms. Some example of instance's metadata could be: o lifecycle:cold-migratable=true|false o ha:maintenance-strategy:in_place|power_off|migrate * It was discussed that Nova could infer which operations are valid or not, based on information like: virt driver, flavor, image properties, etc. This feature was initially named 'instance capabilities' and will require a spec for further discussions. * Another topic of interest, also raised by Sean, was about adding new standard traits to resource providers, like PRESSURE_CPU and PRESSURE_DISK. These traits can be used to weight hosts when placing new VMs. Watcher and the libvirt driver could work on annotating them, but the team generally agreed that the libvirt driver is preferred here. * More info at Nova PTG etherpad [0] and sean's summary blog [1]

[0] https://etherpad.opendev.org/p/r.bf5f1185e201e31ed8c3adeb45e3cf6d [1] https://www.seanmooney.info/blog/2025.2-ptg/#watcher-topics

Please let me know if I missed something. Thanks!

-- Douglas Viroel - dviroel

watcher, aodh, and cloud kitty i believe all have some level if support for Prometheus but they can also use other backends. in not sure what level of enablement they have in osa.

So I think we will test them both with Gnocchi as a backend as of today. I believe the operator is able to flip the backend, but must have a pre-configured prometheus, as Prometheus deployment is out of OSA scope as of today. Having a scrape frontend for Prometheus might be a solution, though I kinda wonder how it's gonna work and where data is gonna be stored in the meanwhile (or in case of networking issues). As still, I believe the timestamp of metrics will be at the moment of scrape from the proxy. So in case Prometheus is not able to reach the proxy - no "billable" data is received. Thus it does not conceptually solve the issue of Prometheus not being the best choice when it comes to some "billing" implementation. As matching of data "issue" timestamp with data "stored" timestamp is crucial there, IMO. чт, 17 апр. 2025 г. в 17:04, <thomas@goirand.fr>:

On Apr 17, 2025 16:49, Sean Mooney <smooney@redhat.com> wrote:

...

On 17/04/2025 13:17, Dmitriy Rabotyagov wrote:

...

...

well gnocchi is also not a native OpenStack telemetry datastore, it

left

...

...

...

our community to pursue its own goals and is now a third party datastore just like Grafana or Prometheus. Yeah, well, true. Is still somehow treated as the "default" thing with Telemetry, likely due to existing integration with Keystone and multi-tenancy support. And beyond it - all other options become opinionated too fast - ie, some do OpenTelemetry, some do Zabbix, VictoriaMetrics, etc. As pretty much from what I got as well, is that still relies on Ceilometer metrics? And then Prometheus is obviously not the best storage for them, as it requires to have pushgatgeway, and afaik prometheus maintainers are strictly against "push" concept to it and treat it as conceptually wrong (on contrary to OpenTelemetry).

i dont know the detail but i know there is work planned for native supprot of Prometheus scrpe endpoint in ceilometer

That is what the telemetry folks discussed during the last online PTG yes. If I understand correctly, the major issue is that prometheus does not support multi-tenancy. Telemetry team hope to add the feature by writing a proxy in front of it.

This would be nice, as Prometheus performs a way better than Gnocchi (it can eat 100 times more metrics per seconds).

Cheers,

Thomas Goirand (zigo)