I have a running instance and I would like to retrieve some statistics on it like cpu usage, memory usage, packets received and sent. ________________________________ Da: Rafael Weingärtner rafaelweingartner@gmail.com Inviato: venerdì 4 marzo 2022 17:28 A: Molka Gharbaoui molka.gharbaoui@santannapisa.it Cc: Openstack Discuss List openstack-discuss@lists.openstack.org Oggetto: Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

What do you or what would you expect to see in Gnocchi being pushed by Ceilometer? What are the resources (volumes, VMs, and so) that you created in the OpenStack environment in DevStack?

On Fri, Mar 4, 2022 at 1:26 PM Molka Gharbaoui <Molka.Gharbaoui@santannapisa.itmailto:Molka.Gharbaoui@santannapisa.it> wrote: Unfortunately, the local.conf file shared by Matthias gives the same result: no metrics are pushed to gnocchi.

Gnocchi database is empty: I can create metrics and resources manually but nothing is pushed from ceilometer to the database. So I guess the problem is at the ceilometer level.

Here is an extract of the ceilometer-polling logs:

2022-03-04 16:13:07.904 679128 INFO ceilometer.polling.manager [-] Polling pollster disk.device.usage in the context of all_pollsters 2022-03-04 16:13:07.904 679128 DEBUG ceilometer.compute.pollsters [-] afcf32c4-e7c2-40fd-8f75-fdf0bfdb5ef9/disk.device.usage volume: 5368709120 _stats_to_sample /opt/stack/ceilometer/ceilometer/compute/pollsters/__init__.py:108 2022-03-04 16:13:07.907 679128 INFO ceilometer.polling.manager [-] Polling pollster disk.device.read.requests in the context of all_pollsters 2022-03-04 16:13:07.908 679128 DEBUG ceilometer.compute.pollsters [-] afcf32c4-e7c2-40fd-8f75-fdf0bfdb5ef9/disk.device.read.requests volume: 6138 _stats_to_sample /opt/stack/ceilometer/ceilometer/compute/pollsters/__init__.py:108 2022-03-04 16:13:08.038 679128 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found. 2022-03-04 16:13:08.038 679128 DEBUG ceilometer.polling.manager [-] Skip pollster network.services.lb.pool, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.162 679128 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found. 2022-03-04 16:13:08.162 679128 DEBUG ceilometer.polling.manager [-] Skip pollster network.services.lb.vip, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.163 679128 DEBUG ceilometer.polling.manager [-] Skip pollster network.services.lb.outgoing.bytes, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.164 679128 ERROR ceilometer.polling.manager [-] Skipping tenant, keystone issue: admin endpoint for identity service in RegionOne region not found: keystoneauth1.exceptions.catalog.EndpointNotFound: admin endpoint for identity service in RegionOne region not found 2022-03-04 16:13:08.164 679128 DEBUG ceilometer.polling.manager [-] Skip pollster radosgw.containers.objects.size, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.164 679128 DEBUG ceilometer.polling.manager [-] Skip pollster port.receive.packets, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.164 679128 DEBUG ceilometer.polling.manager [-] Skip pollster hardware.network.ip.outgoing.datagrams, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.355 679128 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found. 2022-03-04 16:13:08.355 679128 DEBUG ceilometer.polling.manager [-] Skip pollster network.services.lb.listener, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177

And here I report the content of the ceilometer.conf file: [DEFAULT] debug = True transport_url = rabbit://stackrabbit:admin@10.30.2.194:5672/http://stackrabbit:admin@10.30.2.194:5672/

[oslo_messaging_notifications] topics = notifications

[coordination] backend_url = redis://localhost:6379

[notification] workers = 2

[cache] backend_argument = url:redis://localhost:6379 backend_argument = distributed_lock:True backend_argument = db:0 backend_argument = redis_expiration_time:600 backend = dogpile.cache.redis enabled = True

[service_credentials] auth_url = http://10.30.2.194/identityhttp://10.30.2.194/identity/v3/ region_name = RegionOne password = admin username = ceilometer project_name = service project_domain_name = default user_domain_id = default auth_type = password

[keystone_authtoken] memcached_servers = localhost:11211 cafile = /opt/stack/data/ca-bundle.pem project_domain_name = default project_name = admin user_domain_name = default password = admin username = ceilometer auth_url = http://10.30.2.194/identityhttp://10.30.2.194/identity/v3/ interface = public auth_type = password

I believe nothing should be changed in the configuration files since Devstack should configure every components automatically. I also checked pipeline.yaml and polling.yaml files and they seem correct.

Do you have any idea on how to make ceilometer push metrics to the gnocchi database??

Thanks.

________________________________ Da: Matthias Runge <mrunge@matthias-runge.demailto:mrunge@matthias-runge.de> Inviato: giovedì 3 marzo 2022 13:58 A: Molka Gharbaoui <Molka.Gharbaoui@santannapisa.itmailto:Molka.Gharbaoui@santannapisa.it> Oggetto: Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

On 3/3/22 12:31, Molka Gharbaoui wrote:

Hi Matthias,

Sorry for the confusion. Gnocchi is running and I am able to execute commands like "gnocchi metric list" and "gnocchi resource list" without errors even though the response is empty. Also checking mysql, the gnocchi database is correctly created.

By the link I previously shared I wanted just to highlight that there are some differences between what was written there and between the environemnt I installed. I was searching there because I would have expected that Devstack installs and configures every component in automatic with no or minimum external intervention (in fact if I don't consider the Telemetry aspect, the rest works perfectly fine).

I will follow Rafael guidelines to better undrstand where is the issue.

Thanks.

Something I have used in the past is a local.conf like this:

[[local|localrc]] ADMIN_PASSWORD=secret DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD

IMAGE_URL_SITE="https://download.fedoraproject.org" IMAGE_URL_PATH="/pub/fedora/linux/releases/33/Cloud/x86_64/images/" IMAGE_URL_FILE="Fedora-Cloud-Base-33-1.2.x86_64.qcow2" IMAGE_URLS+=","$IMAGE_URL_SITE$IMAGE_URL_PATH$IMAGE_URL_FILE

CEILOMETER_BACKEND=gnocchi USE_PYTHON3="True"

VERBOSE="True"

VERBOSE_NO_TIMESTAMP="True"

# LIBS_FROM_GIT=aodh,ceilometer,cinder,devstack,devstack-gate,dib-utils,diskimage-builder,glance,heat,keystone,neutron,nova,placement,requirements,swift

enable_plugin aodh https://opendev.org/openstack/aodh enable_plugin ceilometer https://opendev.org/openstack/ceilometer enable_plugin heat https://opendev.org/openstack/heat

INSTALL_TEMPEST=False

Hope that helps. The IMAGE_URL_FILE needs probably adjustment, fedora 33 is not supported anymore.

Matthias

---

*Da:* Matthias Runge <mrunge@matthias-runge.demailto:mrunge@matthias-runge.de> *Inviato:* giovedì 3 marzo 2022 11:59 *A:* openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> *Oggetto:* Re: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi On 3/3/22 10:29, Molka Gharbaoui wrote:

...

Thank you Matthias for your reply.

Unfortunately, from the logs I still cannot understand what is the issue.

Could someone kindly indicate me how ceilometer and gnocchi should be configured to get the data?

For example in this link: https://docs.openstack.org/ceilometer/latest/install/install-base-ubuntu.htm...

https://docs.openstack.org/ceilometer/latest/install/install-base-ubuntu.html

...

<https://docs.openstack.org/ceilometer/latest/install/install-base-ubuntu.htm...

https://docs.openstack.org/ceilometer/latest/install/install-base-ubuntu.html> the

...

endpoints should use the following URL: http://controller:8041 http://controller:8041 <http://controller:8041 http://controller:8041> while in my case the

8041 port is not even

...

listening and the API indicated in the dashboard for Gnocchi is http:// <http://controller:8041 http://controller:8041>x.x.x.x/metric.

Thank you in advance.

OpenStack configuration is complex. The guide does not remotely touch on production environments. Please do not confuse devstack with production envionments. Also the doc you linked is not for devstack itself.

The flow in general is: data is generated by the services like nova or glance. Ceilometer listens on the bus and also polls for information. The info is sent then over (as configured in publishers.yaml) to gnocchi. Gnocchi has an api, ceilometer has not. Reading your response, this makes me suspicious if there is gnocchi running. From the initial question it suggests it is, from your most recent response, it is not.

Matthias

-- Rafael Weingärtner

Yes, all ceilometer services are up and running: ceilometer-agent, ceilometer-notification. Actually, all devstack services are up and running. ________________________________ Da: Rafael Weingärtner rafaelweingartner@gmail.com Inviato: venerdì 4 marzo 2022 17:56 A: Molka Gharbaoui molka.gharbaoui@santannapisa.it Cc: Openstack Discuss List openstack-discuss@lists.openstack.org Oggetto: Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

I see. Is the Ceilometer compute agent up and running?

On Fri, Mar 4, 2022 at 1:54 PM Molka Gharbaoui <Molka.Gharbaoui@santannapisa.itmailto:Molka.Gharbaoui@santannapisa.it> wrote: I have a running instance and I would like to retrieve some statistics on it like cpu usage, memory usage, packets received and sent. ________________________________ Da: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Inviato: venerdì 4 marzo 2022 17:28 A: Molka Gharbaoui <molka.gharbaoui@santannapisa.itmailto:molka.gharbaoui@santannapisa.it> Cc: Openstack Discuss List <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Oggetto: Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

What do you or what would you expect to see in Gnocchi being pushed by Ceilometer? What are the resources (volumes, VMs, and so) that you created in the OpenStack environment in DevStack?

On Fri, Mar 4, 2022 at 1:26 PM Molka Gharbaoui <Molka.Gharbaoui@santannapisa.itmailto:Molka.Gharbaoui@santannapisa.it> wrote: Unfortunately, the local.conf file shared by Matthias gives the same result: no metrics are pushed to gnocchi.

Gnocchi database is empty: I can create metrics and resources manually but nothing is pushed from ceilometer to the database. So I guess the problem is at the ceilometer level.

Here is an extract of the ceilometer-polling logs:

2022-03-04 16:13:07.904 679128 INFO ceilometer.polling.manager [-] Polling pollster disk.device.usage in the context of all_pollsters 2022-03-04 16:13:07.904 679128 DEBUG ceilometer.compute.pollsters [-] afcf32c4-e7c2-40fd-8f75-fdf0bfdb5ef9/disk.device.usage volume: 5368709120 _stats_to_sample /opt/stack/ceilometer/ceilometer/compute/pollsters/__init__.py:108 2022-03-04 16:13:07.907 679128 INFO ceilometer.polling.manager [-] Polling pollster disk.device.read.requests in the context of all_pollsters 2022-03-04 16:13:07.908 679128 DEBUG ceilometer.compute.pollsters [-] afcf32c4-e7c2-40fd-8f75-fdf0bfdb5ef9/disk.device.read.requests volume: 6138 _stats_to_sample /opt/stack/ceilometer/ceilometer/compute/pollsters/__init__.py:108 2022-03-04 16:13:08.038 679128 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found. 2022-03-04 16:13:08.038 679128 DEBUG ceilometer.polling.manager [-] Skip pollster network.services.lb.pool, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.162 679128 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found. 2022-03-04 16:13:08.162 679128 DEBUG ceilometer.polling.manager [-] Skip pollster network.services.lb.vip, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.163 679128 DEBUG ceilometer.polling.manager [-] Skip pollster network.services.lb.outgoing.bytes, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.164 679128 ERROR ceilometer.polling.manager [-] Skipping tenant, keystone issue: admin endpoint for identity service in RegionOne region not found: keystoneauth1.exceptions.catalog.EndpointNotFound: admin endpoint for identity service in RegionOne region not found 2022-03-04 16:13:08.164 679128 DEBUG ceilometer.polling.manager [-] Skip pollster radosgw.containers.objects.size, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.164 679128 DEBUG ceilometer.polling.manager [-] Skip pollster port.receive.packets, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.164 679128 DEBUG ceilometer.polling.manager [-] Skip pollster hardware.network.ip.outgoing.datagrams, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.355 679128 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found. 2022-03-04 16:13:08.355 679128 DEBUG ceilometer.polling.manager [-] Skip pollster network.services.lb.listener, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177

And here I report the content of the ceilometer.conf file: [DEFAULT] debug = True transport_url = rabbit://stackrabbit:admin@10.30.2.194:5672/http://stackrabbit:admin@10.30.2.194:5672/

[oslo_messaging_notifications] topics = notifications

[coordination] backend_url = redis://localhost:6379

[notification] workers = 2

[cache] backend_argument = url:redis://localhost:6379 backend_argument = distributed_lock:True backend_argument = db:0 backend_argument = redis_expiration_time:600 backend = dogpile.cache.redis enabled = True

[service_credentials] auth_url = http://10.30.2.194/identityhttp://10.30.2.194/identity/v3/ region_name = RegionOne password = admin username = ceilometer project_name = service project_domain_name = default user_domain_id = default auth_type = password

[keystone_authtoken] memcached_servers = localhost:11211 cafile = /opt/stack/data/ca-bundle.pem project_domain_name = default project_name = admin user_domain_name = default password = admin username = ceilometer auth_url = http://10.30.2.194/identityhttp://10.30.2.194/identity/v3/ interface = public auth_type = password

I believe nothing should be changed in the configuration files since Devstack should configure every components automatically. I also checked pipeline.yaml and polling.yaml files and they seem correct.

Do you have any idea on how to make ceilometer push metrics to the gnocchi database??

Thanks.

________________________________ Da: Matthias Runge <mrunge@matthias-runge.demailto:mrunge@matthias-runge.de> Inviato: giovedì 3 marzo 2022 13:58 A: Molka Gharbaoui <Molka.Gharbaoui@santannapisa.itmailto:Molka.Gharbaoui@santannapisa.it> Oggetto: Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

On 3/3/22 12:31, Molka Gharbaoui wrote:

Hi Matthias,

Sorry for the confusion. Gnocchi is running and I am able to execute commands like "gnocchi metric list" and "gnocchi resource list" without errors even though the response is empty. Also checking mysql, the gnocchi database is correctly created.

By the link I previously shared I wanted just to highlight that there are some differences between what was written there and between the environemnt I installed. I was searching there because I would have expected that Devstack installs and configures every component in automatic with no or minimum external intervention (in fact if I don't consider the Telemetry aspect, the rest works perfectly fine).

I will follow Rafael guidelines to better undrstand where is the issue.

Thanks.

Something I have used in the past is a local.conf like this:

[[local|localrc]] ADMIN_PASSWORD=secret DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD

IMAGE_URL_SITE="https://download.fedoraproject.org" IMAGE_URL_PATH="/pub/fedora/linux/releases/33/Cloud/x86_64/images/" IMAGE_URL_FILE="Fedora-Cloud-Base-33-1.2.x86_64.qcow2" IMAGE_URLS+=","$IMAGE_URL_SITE$IMAGE_URL_PATH$IMAGE_URL_FILE

CEILOMETER_BACKEND=gnocchi USE_PYTHON3="True"

VERBOSE="True"

VERBOSE_NO_TIMESTAMP="True"

# LIBS_FROM_GIT=aodh,ceilometer,cinder,devstack,devstack-gate,dib-utils,diskimage-builder,glance,heat,keystone,neutron,nova,placement,requirements,swift

enable_plugin aodh https://opendev.org/openstack/aodh enable_plugin ceilometer https://opendev.org/openstack/ceilometer enable_plugin heat https://opendev.org/openstack/heat

INSTALL_TEMPEST=False

Hope that helps. The IMAGE_URL_FILE needs probably adjustment, fedora 33 is not supported anymore.

Matthias

---

*Da:* Matthias Runge <mrunge@matthias-runge.demailto:mrunge@matthias-runge.de> *Inviato:* giovedì 3 marzo 2022 11:59 *A:* openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> *Oggetto:* Re: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi On 3/3/22 10:29, Molka Gharbaoui wrote:

...

Thank you Matthias for your reply.

Unfortunately, from the logs I still cannot understand what is the issue.

Could someone kindly indicate me how ceilometer and gnocchi should be configured to get the data?

For example in this link: https://docs.openstack.org/ceilometer/latest/install/install-base-ubuntu.htm...

https://docs.openstack.org/ceilometer/latest/install/install-base-ubuntu.html

...

<https://docs.openstack.org/ceilometer/latest/install/install-base-ubuntu.htm...

https://docs.openstack.org/ceilometer/latest/install/install-base-ubuntu.html> the

...

endpoints should use the following URL: http://controller:8041 http://controller:8041 <http://controller:8041 http://controller:8041> while in my case the

8041 port is not even

...

listening and the API indicated in the dashboard for Gnocchi is http:// <http://controller:8041 http://controller:8041>x.x.x.x/metric.

Thank you in advance.

OpenStack configuration is complex. The guide does not remotely touch on production environments. Please do not confuse devstack with production envionments. Also the doc you linked is not for devstack itself.

The flow in general is: data is generated by the services like nova or glance. Ceilometer listens on the bus and also polls for information. The info is sent then over (as configured in publishers.yaml) to gnocchi. Gnocchi has an api, ceilometer has not. Reading your response, this makes me suspicious if there is gnocchi running. From the initial question it suggests it is, from your most recent response, it is not.

Matthias

-- Rafael Weingärtner

-- Rafael Weingärtner

Yes sorry, I forgot to mention it but it is also running. ________________________________ Da: Rafael Weingärtner rafaelweingartner@gmail.com Inviato: venerdì 4 marzo 2022 19:48 A: Molka Gharbaoui molka.gharbaoui@santannapisa.it Cc: Openstack Discuss List openstack-discuss@lists.openstack.org Oggetto: Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

And, what about ceilometer-compute?

On Fri, Mar 4, 2022 at 2:04 PM Molka Gharbaoui <Molka.Gharbaoui@santannapisa.itmailto:Molka.Gharbaoui@santannapisa.it> wrote: Yes, all ceilometer services are up and running: ceilometer-agent, ceilometer-notification. Actually, all devstack services are up and running. ________________________________ Da: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Inviato: venerdì 4 marzo 2022 17:56 A: Molka Gharbaoui <molka.gharbaoui@santannapisa.itmailto:molka.gharbaoui@santannapisa.it> Cc: Openstack Discuss List <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Oggetto: Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

I see. Is the Ceilometer compute agent up and running?

On Fri, Mar 4, 2022 at 1:54 PM Molka Gharbaoui <Molka.Gharbaoui@santannapisa.itmailto:Molka.Gharbaoui@santannapisa.it> wrote: I have a running instance and I would like to retrieve some statistics on it like cpu usage, memory usage, packets received and sent. ________________________________ Da: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Inviato: venerdì 4 marzo 2022 17:28 A: Molka Gharbaoui <molka.gharbaoui@santannapisa.itmailto:molka.gharbaoui@santannapisa.it> Cc: Openstack Discuss List <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Oggetto: Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

What do you or what would you expect to see in Gnocchi being pushed by Ceilometer? What are the resources (volumes, VMs, and so) that you created in the OpenStack environment in DevStack?

On Fri, Mar 4, 2022 at 1:26 PM Molka Gharbaoui <Molka.Gharbaoui@santannapisa.itmailto:Molka.Gharbaoui@santannapisa.it> wrote: Unfortunately, the local.conf file shared by Matthias gives the same result: no metrics are pushed to gnocchi.

Gnocchi database is empty: I can create metrics and resources manually but nothing is pushed from ceilometer to the database. So I guess the problem is at the ceilometer level.

Here is an extract of the ceilometer-polling logs:

2022-03-04 16:13:07.904 679128 INFO ceilometer.polling.manager [-] Polling pollster disk.device.usage in the context of all_pollsters 2022-03-04 16:13:07.904 679128 DEBUG ceilometer.compute.pollsters [-] afcf32c4-e7c2-40fd-8f75-fdf0bfdb5ef9/disk.device.usage volume: 5368709120 _stats_to_sample /opt/stack/ceilometer/ceilometer/compute/pollsters/__init__.py:108 2022-03-04 16:13:07.907 679128 INFO ceilometer.polling.manager [-] Polling pollster disk.device.read.requests in the context of all_pollsters 2022-03-04 16:13:07.908 679128 DEBUG ceilometer.compute.pollsters [-] afcf32c4-e7c2-40fd-8f75-fdf0bfdb5ef9/disk.device.read.requests volume: 6138 _stats_to_sample /opt/stack/ceilometer/ceilometer/compute/pollsters/__init__.py:108 2022-03-04 16:13:08.038 679128 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found. 2022-03-04 16:13:08.038 679128 DEBUG ceilometer.polling.manager [-] Skip pollster network.services.lb.pool, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.162 679128 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found. 2022-03-04 16:13:08.162 679128 DEBUG ceilometer.polling.manager [-] Skip pollster network.services.lb.vip, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.163 679128 DEBUG ceilometer.polling.manager [-] Skip pollster network.services.lb.outgoing.bytes, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.164 679128 ERROR ceilometer.polling.manager [-] Skipping tenant, keystone issue: admin endpoint for identity service in RegionOne region not found: keystoneauth1.exceptions.catalog.EndpointNotFound: admin endpoint for identity service in RegionOne region not found 2022-03-04 16:13:08.164 679128 DEBUG ceilometer.polling.manager [-] Skip pollster radosgw.containers.objects.size, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.164 679128 DEBUG ceilometer.polling.manager [-] Skip pollster port.receive.packets, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.164 679128 DEBUG ceilometer.polling.manager [-] Skip pollster hardware.network.ip.outgoing.datagrams, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177 2022-03-04 16:13:08.355 679128 WARNING ceilometer.neutron_client [-] The resource could not be found.: neutronclient.common.exceptions.NotFound: The resource could not be found. 2022-03-04 16:13:08.355 679128 DEBUG ceilometer.polling.manager [-] Skip pollster network.services.lb.listener, no resources found this cycle poll_and_notify /opt/stack/ceilometer/ceilometer/polling/manager.py:177

And here I report the content of the ceilometer.conf file: [DEFAULT] debug = True transport_url = rabbit://stackrabbit:admin@10.30.2.194:5672/http://stackrabbit:admin@10.30.2.194:5672/

[oslo_messaging_notifications] topics = notifications

[coordination] backend_url = redis://localhost:6379

[notification] workers = 2

[cache] backend_argument = url:redis://localhost:6379 backend_argument = distributed_lock:True backend_argument = db:0 backend_argument = redis_expiration_time:600 backend = dogpile.cache.redis enabled = True

[service_credentials] auth_url = http://10.30.2.194/identityhttp://10.30.2.194/identity/v3/ region_name = RegionOne password = admin username = ceilometer project_name = service project_domain_name = default user_domain_id = default auth_type = password

[keystone_authtoken] memcached_servers = localhost:11211 cafile = /opt/stack/data/ca-bundle.pem project_domain_name = default project_name = admin user_domain_name = default password = admin username = ceilometer auth_url = http://10.30.2.194/identityhttp://10.30.2.194/identity/v3/ interface = public auth_type = password

I believe nothing should be changed in the configuration files since Devstack should configure every components automatically. I also checked pipeline.yaml and polling.yaml files and they seem correct.

Do you have any idea on how to make ceilometer push metrics to the gnocchi database??

Thanks.

________________________________ Da: Matthias Runge <mrunge@matthias-runge.demailto:mrunge@matthias-runge.de> Inviato: giovedì 3 marzo 2022 13:58 A: Molka Gharbaoui <Molka.Gharbaoui@santannapisa.itmailto:Molka.Gharbaoui@santannapisa.it> Oggetto: Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

On 3/3/22 12:31, Molka Gharbaoui wrote:

Hi Matthias,

Sorry for the confusion. Gnocchi is running and I am able to execute commands like "gnocchi metric list" and "gnocchi resource list" without errors even though the response is empty. Also checking mysql, the gnocchi database is correctly created.

By the link I previously shared I wanted just to highlight that there are some differences between what was written there and between the environemnt I installed. I was searching there because I would have expected that Devstack installs and configures every component in automatic with no or minimum external intervention (in fact if I don't consider the Telemetry aspect, the rest works perfectly fine).

I will follow Rafael guidelines to better undrstand where is the issue.

Thanks.

Something I have used in the past is a local.conf like this:

[[local|localrc]] ADMIN_PASSWORD=secret DATABASE_PASSWORD=$ADMIN_PASSWORD RABBIT_PASSWORD=$ADMIN_PASSWORD SERVICE_PASSWORD=$ADMIN_PASSWORD

IMAGE_URL_SITE="https://download.fedoraproject.org" IMAGE_URL_PATH="/pub/fedora/linux/releases/33/Cloud/x86_64/images/" IMAGE_URL_FILE="Fedora-Cloud-Base-33-1.2.x86_64.qcow2" IMAGE_URLS+=","$IMAGE_URL_SITE$IMAGE_URL_PATH$IMAGE_URL_FILE

CEILOMETER_BACKEND=gnocchi USE_PYTHON3="True"

VERBOSE="True"

VERBOSE_NO_TIMESTAMP="True"

# LIBS_FROM_GIT=aodh,ceilometer,cinder,devstack,devstack-gate,dib-utils,diskimage-builder,glance,heat,keystone,neutron,nova,placement,requirements,swift

enable_plugin aodh https://opendev.org/openstack/aodh enable_plugin ceilometer https://opendev.org/openstack/ceilometer enable_plugin heat https://opendev.org/openstack/heat

INSTALL_TEMPEST=False

Hope that helps. The IMAGE_URL_FILE needs probably adjustment, fedora 33 is not supported anymore.

Matthias

---

*Da:* Matthias Runge <mrunge@matthias-runge.demailto:mrunge@matthias-runge.de> *Inviato:* giovedì 3 marzo 2022 11:59 *A:* openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> *Oggetto:* Re: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi On 3/3/22 10:29, Molka Gharbaoui wrote:

...

Thank you Matthias for your reply.

Unfortunately, from the logs I still cannot understand what is the issue.

Could someone kindly indicate me how ceilometer and gnocchi should be configured to get the data?

For example in this link: https://docs.openstack.org/ceilometer/latest/install/install-base-ubuntu.htm...

https://docs.openstack.org/ceilometer/latest/install/install-base-ubuntu.html

...

<https://docs.openstack.org/ceilometer/latest/install/install-base-ubuntu.htm...

https://docs.openstack.org/ceilometer/latest/install/install-base-ubuntu.html> the

...

endpoints should use the following URL: http://controller:8041 http://controller:8041 <http://controller:8041 http://controller:8041> while in my case the

8041 port is not even

...

listening and the API indicated in the dashboard for Gnocchi is http:// <http://controller:8041 http://controller:8041>x.x.x.x/metric.

Thank you in advance.

OpenStack configuration is complex. The guide does not remotely touch on production environments. Please do not confuse devstack with production envionments. Also the doc you linked is not for devstack itself.

The flow in general is: data is generated by the services like nova or glance. Ceilometer listens on the bus and also polls for information. The info is sent then over (as configured in publishers.yaml) to gnocchi. Gnocchi has an api, ceilometer has not. Reading your response, this makes me suspicious if there is gnocchi running. From the initial question it suggests it is, from your most recent response, it is not.

Matthias

-- Rafael Weingärtner

-- Rafael Weingärtner

-- Rafael Weingärtner

No errors in the gnocchi log files.

Here is part of the output of the "gnocchi metric list --debug":

http://10.30.2.194:80 "GET /metric/v1/metric HTTP/1.1" 200 2 RESP: [200] Connection: close Content-Length: 2 Content-Type: application/json Date: Sat, 05 Mar 2022 09:33:24 GMT Server: Apache/2.4.41 (Ubuntu) RESP BODY: []

The response body is empty since the database is empty. Other output from the log files:

[697660] DEBUG gnocchi.chef: Processing measures for sack incoming128-123 2022-03-05 10:17:59,042 [697663] DEBUG gnocchi.chef: Processing measures for sack incoming128-23 [697660] DEBUG gnocchi.cli.metricd: Full scan of sacks has been done

There is only one error in all the log files and it regards ceilometer logs. I report it here:

2022-03-04 16:13:08.164 679128 ERROR ceilometer.polling.manager [-] Skipping tenant, keystone issue: admin endpoint for identity service in RegionOne region not found: keystoneauth1.exceptions.catalog.EndpointNotFound: admin endpoint for identity service in RegionOne region not found

Thank you for your help!

________________________________ Da: Rafael Weingärtner rafaelweingartner@gmail.com Inviato: venerdì 4 marzo 2022 21:54 A: Molka Gharbaoui molka.gharbaoui@santannapisa.it Cc: Openstack Discuss List openstack-discuss@lists.openstack.org Oggetto: Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

Did you check Gnocchi log files? Is there some error or something happening there?

-- Rafael Weingärtner

No, the message: "%d measures posted against %d metrics through %d resources" does not appear in ceilometer logs.

I executed: ceilometer-upgrade, ceilometer-polling and ceilometer-agent-notification commands.

________________________________ Da: Rafael Weingärtner rafaelweingartner@gmail.com Inviato: sabato 5 marzo 2022 16:00 A: Molka Gharbaoui molka.gharbaoui@santannapisa.it Cc: Openstack Discuss List openstack-discuss@lists.openstack.org Oggetto: Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

The messages: "%d measures posted against %d metrics through %d resources" appear in the log files of Ceilometer?

On Sat, Mar 5, 2022 at 11:12 AM Molka Gharbaoui <Molka.Gharbaoui@santannapisa.itmailto:Molka.Gharbaoui@santannapisa.it> wrote: No errors in the gnocchi log files.

Here is part of the output of the "gnocchi metric list --debug":

http://10.30.2.194:80 "GET /metric/v1/metric HTTP/1.1" 200 2 RESP: [200] Connection: close Content-Length: 2 Content-Type: application/json Date: Sat, 05 Mar 2022 09:33:24 GMT Server: Apache/2.4.41 (Ubuntu) RESP BODY: []

The response body is empty since the database is empty. Other output from the log files:

[697660] DEBUG gnocchi.chef: Processing measures for sack incoming128-123 2022-03-05 10:17:59,042 [697663] DEBUG gnocchi.chef: Processing measures for sack incoming128-23 [697660] DEBUG gnocchi.cli.metricd: Full scan of sacks has been done

There is only one error in all the log files and it regards ceilometer logs. I report it here:

2022-03-04 16:13:08.164 679128 ERROR ceilometer.polling.manager [-] Skipping tenant, keystone issue: admin endpoint for identity service in RegionOne region not found: keystoneauth1.exceptions.catalog.EndpointNotFound: admin endpoint for identity service in RegionOne region not found

Thank you for your help!

________________________________ Da: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Inviato: venerdì 4 marzo 2022 21:54 A: Molka Gharbaoui <molka.gharbaoui@santannapisa.itmailto:molka.gharbaoui@santannapisa.it> Cc: Openstack Discuss List <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Oggetto: Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

Did you check Gnocchi log files? Is there some error or something happening there?

-- Rafael Weingärtner

-- Rafael Weingärtner

What does "gnocchi resource show 3ef0373a-558c-4818-b693-7536ee70d283" display?

On Tue, Mar 8, 2022 at 12:00 PM Molka Gharbaoui < Molka.Gharbaoui@santannapisa.it> wrote:

I left the instance running for more than 1 hour and now when I check the logs of "ceilometer-polling" I can see the following messages:

DEBUG ceilometer.compute.pollsters [-] 3ef0373a-558c-4818-b693-7536ee70d283/network.outgoing.bytes volume: 3608 _stats_to_sample /opt/stack/ceilometer/ceilometer/compute/pollsters/__init__.py:108 DEBUG ceilometer.compute.pollsters [-] 3ef0373a-558c-4818-b693-7536ee70d283/network.outgoing.packets volume: 25 _stats_to_sample /opt/stack/ceilometer/ceilometer/compute/pollsters/__init__.py:108 DEBUG ceilometer.compute.pollsters [-] 3ef0373a-558c-4818-b693-7536ee70d283/disk.device.write.requests volume: 4568 _stats_to_sample /opt/stack/ceilometer/ceilometer/compute/pollsters/__init__.py:108

So I guess that Ceilometer is starting to poll some statistics.

The "gnocchi resource list" also now gives the following output: ID Type 3ef0373a-558c-4818-b693-7536ee70d283 | instance

However, the "gnocchi metric list" is still empty.

Looking in the logs of "ceilometer-agent-notification", I found the following error: ERROR ceilometer.pipeline.sample [-] Pipeline meter_sink: Continue after error from publisher <ceilometer.publisher.gnocchi.GnocchiPublisher object at 0x7fc9800f9ee0>: gnocchiclient.exceptions.Conflict: <html>

<head> <title>409 Conflict</title> </head> <body> <h1>409 Conflict</h1> There was a conflict when trying to complete your request.<br /><br /> Archive policy ceilometer-low already exists </body> </html> (HTTP 409)

Do you know how to resolve this?

Thanks.

It seems that Ceilometer is trying to re-create the archive-policy, but it already exists. Can you check the archive policy in Gnocchi?

On Tue, Mar 8, 2022 at 1:02 PM Molka Gharbaoui < Molka.Gharbaoui@santannapisa.it> wrote:

+-----------------------+----------------------------------- --------------------------------+ | Field | Value | +-----------------------+----------------------------------- --------------------------------+ | created_by_project_id | ce87d288bf5846f1b54d3664802fd06d | | created_by_user_id | f8619bc10b584e3d9365ca201d00c9e6 | | creator | f8619bc10b584e3d9365ca201d00c9e6: ce87d288bf5846f1b54d3664802fd06d | | ended_at | None | | id | 3ef0373a-558c-4818-b693-7536ee70d283 | | metrics | | | original_resource_id | 3ef0373a-558c-4818-b693-7536ee70d283 | | project_id | f25dced0e26947dd924fc2c9984282ad | | revision_end | None | | revision_start | 2022-03-08T15:46:36.110838+00:00 | | started_at | 2022-03-08T15:46:36.110825+00:00 | | type | instance | | user_id | 4d36e9b17ff04d55853cddf111a0d4dd | +-----------------------+-----------------------------------

--------------------------------+

*Da:* Rafael Weingärtner rafaelweingartner@gmail.com *Inviato:* martedì 8 marzo 2022 16:56 *A:* Molka Gharbaoui molka.gharbaoui@santannapisa.it *Cc:* Openstack Discuss List openstack-discuss@lists.openstack.org *Oggetto:* Re: R: R: [Devstack][Gnocchi][Ceilometer] Not able to retrieve metrics using ceilometer and gnocchi

What does "gnocchi resource show 3ef0373a-558c-4818-b693-7536ee70d283" display?

On Tue, Mar 8, 2022 at 12:00 PM Molka Gharbaoui < Molka.Gharbaoui@santannapisa.it> wrote:

I left the instance running for more than 1 hour and now when I check the logs of "ceilometer-polling" I can see the following messages:

DEBUG ceilometer.compute.pollsters [-] 3ef0373a-558c-4818-b693-7536ee70d283/network.outgoing.bytes volume: 3608 _stats_to_sample /opt/stack/ceilometer/ceilometer/compute/pollsters/__init__.py:108 DEBUG ceilometer.compute.pollsters [-] 3ef0373a-558c-4818-b693-7536ee70d283/network.outgoing.packets volume: 25 _stats_to_sample /opt/stack/ceilometer/ceilometer/compute/pollsters/__init__.py:108 DEBUG ceilometer.compute.pollsters [-] 3ef0373a-558c-4818-b693-7536ee70d283/disk.device.write.requests volume: 4568 _stats_to_sample /opt/stack/ceilometer/ceilometer/compute/pollsters/__init__.py:108

So I guess that Ceilometer is starting to poll some statistics.

The "gnocchi resource list" also now gives the following output: ID Type 3ef0373a-558c-4818-b693-7536ee70d283 | instance

However, the "gnocchi metric list" is still empty.

Looking in the logs of "ceilometer-agent-notification", I found the following error: ERROR ceilometer.pipeline.sample [-] Pipeline meter_sink: Continue after error from publisher <ceilometer.publisher.gnocchi.GnocchiPublisher object at 0x7fc9800f9ee0>: gnocchiclient.exceptions.Conflict: <html>

<head> <title>409 Conflict</title> </head> <body> <h1>409 Conflict</h1> There was a conflict when trying to complete your request.<br /><br /> Archive policy ceilometer-low already exists </body> </html> (HTTP 409)

Do you know how to resolve this?

Thanks.

-- Rafael Weingärtner