[ceilometer] radosgw pollster
Hi,
I failed to poll any usage data from our radosgw. I get
2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True,
[...]
id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError
Configurations like: cat polling.yaml --- sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size
Also tried radosgw.api.requests instead of radowsgw.usage.
ceilometer.conf [...] [service_types] radosgw = object-store
[rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx
[rgw_client] implicit_tenants = true
Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | public | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s |
Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" }
radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true
Any idea whats going on?
All the best, Florian
Hi Florian,
which version of OpenStack are you using? The radosgw metric names were different in some versions: https://bugs.launchpad.net/ceilometer/+bug/1726458
Christian
Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann < florian.engelmann@everyware.ch>:
Hi,
I failed to poll any usage data from our radosgw. I get
2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True,
[...]
id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError
Configurations like: cat polling.yaml
sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size
Also tried radosgw.api.requests instead of radowsgw.usage.
ceilometer.conf [...] [service_types] radosgw = object-store
[rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx
[rgw_client] implicit_tenants = true
Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | public | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s |
Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" }
radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true
Any idea whats going on?
All the best, Florian
Hi Christian,
Am 2/26/19 um 11:00 AM schrieb Christian Zunker:
Hi Florian,
which version of OpenStack are you using? The radosgw metric names were different in some versions: https://bugs.launchpad.net/ceilometer/+bug/1726458
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. As far as I am able to understand python it looks like the error is happening in polling.manager line 222:
https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manag...
But I do not understand why. I tried to enable debug logging but the error does not log any additional information. The poller is not even trying to reach/poll our RadosGWs. Looks like that manger is blocking those polls.
All the best, Florian
Christian
Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.ch mailto:florian.engelmann@everyware.ch>:
Hi, I failed to poll any usage data from our radosgw. I get 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True, [...] id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError Configurations like: cat polling.yaml --- sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size Also tried radosgw.api.requests instead of radowsgw.usage. ceilometer.conf [...] [service_types] radosgw = object-store [rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx [rgw_client] implicit_tenants = true Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | public | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s | Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" } radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true Any idea whats going on? All the best, Florian
Hi Florian,
have you tried different permissions for your ceilometer user in radosgw? According to the docs you need an admin user: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.ht... Our user has these caps: usage=read,write;metadata=read,write;users=read,write;buckets=read,write
We also had to add the requests-aws pip package to query radosgw from ceilometer: https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.htm...
Christian
Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann < florian.engelmann@everyware.ch>:
Hi Christian,
Am 2/26/19 um 11:00 AM schrieb Christian Zunker:
Hi Florian,
which version of OpenStack are you using? The radosgw metric names were different in some versions: https://bugs.launchpad.net/ceilometer/+bug/1726458
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. As far as I am able to understand python it looks like the error is happening in polling.manager line 222:
https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manag...
But I do not understand why. I tried to enable debug logging but the error does not log any additional information. The poller is not even trying to reach/poll our RadosGWs. Looks like that manger is blocking those polls.
All the best, Florian
Christian
Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch
:
Hi, I failed to poll any usage data from our radosgw. I get 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-]Polling
pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-]Prevent
pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True, [...] id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError Configurations like: cat polling.yaml --- sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size Also tried radosgw.api.requests instead of radowsgw.usage. ceilometer.conf [...] [service_types] radosgw = object-store [rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx [rgw_client] implicit_tenants = true Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True |public
| https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s | Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" } radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true Any idea whats going on? All the best, Florian--
EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich
tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.ch web: http://www.everyware.ch
Hi Christian,
thank you for your feedback and help! Permissions are fine as I tried to poll the Endpoint successfully with curl and the user (key + secret) we created (and is configured in ceilometer.conf).
I saw the requests-aws is used in OSA and it is indeed missing in the kolla container (we use "source" not binary).
https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-...
I will build a new ceilometer container including requests-aws tomorrow to see if this fixes the problem.
All the best,
Florian
________________________________ From: Christian Zunker christian.zunker@codecentric.cloud Sent: Wednesday, February 27, 2019 9:09 AM To: Engelmann Florian Cc: openstack-discuss@lists.openstack.org Subject: Re: [ceilometer] radosgw pollster
Hi Florian,
have you tried different permissions for your ceilometer user in radosgw? According to the docs you need an admin user: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.ht... Our user has these caps: usage=read,write;metadata=read,write;users=read,write;buckets=read,write
We also had to add the requests-aws pip package to query radosgw from ceilometer: https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.htm...
Christian
Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.chmailto:florian.engelmann@everyware.ch>: Hi Christian,
Am 2/26/19 um 11:00 AM schrieb Christian Zunker:
Hi Florian,
which version of OpenStack are you using? The radosgw metric names were different in some versions: https://bugs.launchpad.net/ceilometer/+bug/1726458
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. As far as I am able to understand python it looks like the error is happening in polling.manager line 222:
https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manag...
But I do not understand why. I tried to enable debug logging but the error does not log any additional information. The poller is not even trying to reach/poll our RadosGWs. Looks like that manger is blocking those polls.
All the best, Florian
Christian
Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.chmailto:florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.chmailto:florian.engelmann@everyware.ch>>:
Hi, I failed to poll any usage data from our radosgw. I get 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True, [...] id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError Configurations like: cat polling.yaml --- sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size Also tried radosgw.api.requests instead of radowsgw.usage. ceilometer.conf [...] [service_types] radosgw = object-store [rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx [rgw_client] implicit_tenants = true Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | public | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s | Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" } radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true Any idea whats going on? All the best, Florian
--
EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich
tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.chmailto:florian.engelmann@everyware.ch web: http://www.everyware.ch
Hi Christian,
looks like a hit:
https://github.com/openstack/ceilometer/commit/c9eb2d44df7cafde1294123d66445...
You made my day!
I will test tomorrow and report back!
All the best,
Florian
________________________________ From: Engelmann Florian florian.engelmann@everyware.ch Sent: Wednesday, February 27, 2019 8:33 PM To: Christian Zunker Cc: openstack-discuss@lists.openstack.org Subject: Re: [ceilometer] radosgw pollster
Hi Christian,
thank you for your feedback and help! Permissions are fine as I tried to poll the Endpoint successfully with curl and the user (key + secret) we created (and is configured in ceilometer.conf).
I saw the requests-aws is used in OSA and it is indeed missing in the kolla container (we use "source" not binary).
https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-...
I will build a new ceilometer container including requests-aws tomorrow to see if this fixes the problem.
All the best,
Florian
________________________________ From: Christian Zunker christian.zunker@codecentric.cloud Sent: Wednesday, February 27, 2019 9:09 AM To: Engelmann Florian Cc: openstack-discuss@lists.openstack.org Subject: Re: [ceilometer] radosgw pollster
Hi Florian,
have you tried different permissions for your ceilometer user in radosgw? According to the docs you need an admin user: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.ht... Our user has these caps: usage=read,write;metadata=read,write;users=read,write;buckets=read,write
We also had to add the requests-aws pip package to query radosgw from ceilometer: https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.htm...
Christian
Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.chmailto:florian.engelmann@everyware.ch>: Hi Christian,
Am 2/26/19 um 11:00 AM schrieb Christian Zunker:
Hi Florian,
which version of OpenStack are you using? The radosgw metric names were different in some versions: https://bugs.launchpad.net/ceilometer/+bug/1726458
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. As far as I am able to understand python it looks like the error is happening in polling.manager line 222:
https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manag...
But I do not understand why. I tried to enable debug logging but the error does not log any additional information. The poller is not even trying to reach/poll our RadosGWs. Looks like that manger is blocking those polls.
All the best, Florian
Christian
Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.chmailto:florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.chmailto:florian.engelmann@everyware.ch>>:
Hi, I failed to poll any usage data from our radosgw. I get 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True, [...] id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError Configurations like: cat polling.yaml --- sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size Also tried radosgw.api.requests instead of radowsgw.usage. ceilometer.conf [...] [service_types] radosgw = object-store [rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx [rgw_client] implicit_tenants = true Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | public | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s | Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" } radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true Any idea whats going on? All the best, Florian
--
EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich
tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.chmailto:florian.engelmann@everyware.ch web: http://www.everyware.ch
Hi Christian,
after adding requests-aws to the container the PollsterPermanentError is gone and the ceilometer polling log looks clean to me:
2019-02-28 10:29:46.163 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.167 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.objects in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.172 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.objects.size in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.177 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.objects.containers in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.182 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.usage in the context of radosgw_300s_pollsters
I still do not get any data in gnocchi:
openstack metric resource list -c type -f value | sort -u generic image instance instance_disk instance_network_interface network volume
My RadosGW logs do show the poller requests: 2019-02-28 10:29:54.767266 7f67a550e700 20 HTTP_ACCEPT=*/* 2019-02-28 10:29:54.767272 7f67a550e700 20 HTTP_ACCEPT_ENCODING=gzip, deflate 2019-02-28 10:29:54.767274 7f67a550e700 20 HTTP_AUTHORIZATION=AWS 0APxxxxxxxxx:vxxxxxxxICq/qQXxxxxxxxxxuuc= [...] 2019-02-28 10:29:54.767294 7f67a550e700 20 REMOTE_ADDR=10.xx.xxx.xxx 2019-02-28 10:29:54.767295 7f67a550e700 20 REQUEST_METHOD=GET 2019-02-28 10:29:54.767296 7f67a550e700 20 REQUEST_URI=/admin/usage 2019-02-28 10:29:54.767297 7f67a550e700 20 SCRIPT_URI=/admin/usage [...] in_hosted_domain_s3website=0 s->info.domain=rgw.xxxxxx s->info.request_uri=/admin/usage 2019-02-28 10:29:54.767479 7f67a550e700 10 handler=16RGWHandler_Usage 2019-02-28 10:29:54.767491 7f67a550e700 2 req 215268:0.000183::GET /admin/usage::getting op 0 2019-02-28 10:29:54.767495 7f67a550e700 10 op=15RGWOp_Usage_Get 2019-02-28 10:29:54.767496 7f67a550e700 2 req 215268:0.000196::GET /admin/usage:get_usage:verifying requester 2019-02-28 10:29:54.767528 7f67a550e700 20 rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2019-02-28 10:29:54.767530 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2019-02-28 10:29:54.767558 7f67a550e700 20 rgw::auth::s3::S3AnonymousEngine denied with reason=-1 2019-02-28 10:29:54.767561 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::AWSv2ExternalAuthStrategy 2019-02-28 10:29:54.767573 7f67a550e700 20 rgw::auth::s3::AWSv2ExternalAuthStrategy: trying rgw::auth::keystone::EC2Engine 2019-02-28 10:29:54.767602 7f67a550e700 10 get_canon_resource(): dest=/admin/usage 2019-02-28 10:29:54.767605 7f67a550e700 10 string_to_sign: 2019-02-28 10:29:54.767630 7f67a550e700 20 sending request to https://keystone.xxxxxxxxx/v3/auth/tokens 2019-02-28 10:29:54.767678 7f67a550e700 20 ssl verification is set to off 2019-02-28 10:29:55.194088 7f67a550e700 20 sending request to https://keystone.xxxxxxxx/v3/s3tokens 2019-02-28 10:29:55.194115 7f67a550e700 20 ssl verification is set to off 2019-02-28 10:29:55.226841 7f67a550e700 20 rgw::auth::keystone::EC2Engine denied with reason=-2028 2019-02-28 10:29:55.226852 7f67a550e700 20 rgw::auth::s3::AWSv2ExternalAuthStrategy denied with reason=-2028 2019-02-28 10:29:55.226855 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine 2019-02-28 10:29:55.226878 7f67a550e700 10 get_canon_resource(): dest=/admin/usage 2019-02-28 10:29:55.226881 7f67a550e700 10 string_to_sign: 2019-02-28 10:29:55.226943 7f67a550e700 15 string_to_sign=GET [...] 2019-02-28 10:29:55.226972 7f67a550e700 15 compare=0 2019-02-28 10:29:55.227099 7f67a550e700 20 rgw::auth::s3::LocalEngine granted access 2019-02-28 10:29:55.227103 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy granted access 2019-02-28 10:29:55.227106 7f67a550e700 2 req 215268:0.459806::GET /admin/usage:get_usage:normalizing buckets and tenants 2019-02-28 10:29:55.227109 7f67a550e700 2 req 215268:0.459809::GET /admin/usage:get_usage:init permissions 2019-02-28 10:29:55.227136 7f67a550e700 2 req 215268:0.459826::GET /admin/usage:get_usage:recalculating target 2019-02-28 10:29:55.227140 7f67a550e700 2 req 215268:0.459841::GET /admin/usage:get_usage:reading permissions 2019-02-28 10:29:55.227142 7f67a550e700 2 req 215268:0.459842::GET /admin/usage:get_usage:init op 2019-02-28 10:29:55.227143 7f67a550e700 2 req 215268:0.459844::GET /admin/usage:get_usage:verifying op mask 2019-02-28 10:29:55.227145 7f67a550e700 20 required_mask= 0 user.op_mask=7 2019-02-28 10:29:55.227146 7f67a550e700 2 req 215268:0.459846::GET /admin/usage:get_usage:verifying op permissions 2019-02-28 10:29:55.227148 7f67a550e700 2 req 215268:0.459848::GET /admin/usage:get_usage:verifying op params 2019-02-28 10:29:55.227149 7f67a550e700 2 req 215268:0.459850::GET /admin/usage:get_usage:pre-executing 2019-02-28 10:29:55.227150 7f67a550e700 2 req 215268:0.459851::GET /admin/usage:get_usage:executing 2019-02-28 10:29:55.232830 7f67a550e700 2 req 215268:0.465530::GET /admin/usage:get_usage:completing 2019-02-28 10:29:55.232858 7f67a550e700 2 req 215268:0.465559::GET /admin/usage:get_usage:op status=0 2019-02-28 10:29:55.232863 7f67a550e700 2 req 215268:0.465564::GET /admin/usage:get_usage:http status=200 2019-02-28 10:29:55.232865 7f67a550e700 1 ====== req done req=0x7f67a55080a0 op status=0 http_status=200 ====== 2019-02-28 10:29:55.232894 7f67a550e700 1 civetweb: 0x55ced7bd2000: 10.0.81.59 - - [28/Feb/2019:10:29:54 +0100] "GET /admin/usage?uid=2534a3e876ee41f088098fxxxxxxxxx%242534a3e876ee41f088098f53xxxxxxx HTTP/1.1" 200 0 - python-requests/2.19.1
Using curl to do the same request gives me the correct usage results: {"entries":[{"user":"a772e4abxxxxxxxx4559d$a772e4ab888exxxxxxxf4559d","buckets":[{"bucket":"","time":"2019-02-26 12:00:00.000000Z","epoch":1551182400,"owner":"a772e4ab88xxxxxxx88e4f039b3430d688f4559d","categories":[{"category":"list_buckets","bytes_sent":36,"bytes_received":0,"ops":3,"successful_ops":0}]},{"bucket":"-","time":"2019-02-17 15:00:00.000000Z","epoch":1550415600,"owner":"a772e4ab88xxxxxxxa772e4ab888e4f039b3430d688f4559d","categories":[{"category":"get_obj","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":0},{"category":"list_bucket","bytes_sent":132,"bytes_received":0,"ops":1,"successful_ops":0}]},{"bucket":"info","time":"2019-02-26 12:00:00.000000Z","epoch":1551182400,"owner":"a772e4ab888e4f039xxxxxx72e4ab888e4f039b3430d688f4559d","categories":[{"category":"RGWMovedPermanently","bytes_sent":0,"bytes_received":0,"ops":3,"successful_ops":3}]},{"bucket":"test","time":"2019-02-17 15:00:00.000000Z","epoch":1550415600,"owner":"a772e4ab8xxxxxxd$a772e4ab888e4f039b3430d688f4559d","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"list_bucket","bytes_sent":1368,"bytes_received":0,"ops":18,"successful_ops":18},{"category":"put_obj","bytes_sent":0,"bytes_received":10,"ops":1,"successful_ops":1}]}]}],"summary":[{"user":"a772e4abxxxx4f0xxx8f4559d$a772e4ab88xxxxxx30d688f4559d","categories":[{"category":"RGWMovedPermanently","bytes_sent":0,"bytes_received":0,"ops":3,"successful_ops":3},{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"get_obj","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":0},{"category":"list_bucket","bytes_sent":1500,"bytes_received":0,"ops":19,"successful_ops":18},{"category":"list_buckets","bytes_sent":36,"bytes_received":0,"ops":3,"successful_ops":0},{"category":"put_obj","bytes_sent":0,"bytes_received":10,"ops":1,"successful_ops":1}],"total":{"bytes_sent":153* Connection #0 to host 10.xxx.xxx.xxx left intact
My custom archive policy (custom_gnocchi_resources.yaml) looks like: [...] - resource_type: ceph_account metrics: radosgw.objects: radosgw.objects.size: radosgw.objects.containers: radosgw.api.request: radosgw.containers.objects: radosgw.containers.objects.size: [...]
and the pipeline: [...] sources: - name: meter_source meters: - "*" sinks: - meter_sink [...] sinks: - name: meter_sink transformers: publishers: - gnocchi://?resources_definition_file=%2Fetc%2Fceilometer%2Fcustom_gnocchi_resources.yaml [...]
Anything wrong with my archive policy or the pipeline?
All the best, Florian
Am 2/27/19 um 8:38 PM schrieb Engelmann Florian:
Hi Christian,
looks like a hit:
https://github.com/openstack/ceilometer/commit/c9eb2d44df7cafde1294123d66445...
You made my day!
I will test tomorrow and report back!
All the best,
Florian
*From:* Engelmann Florian florian.engelmann@everyware.ch *Sent:* Wednesday, February 27, 2019 8:33 PM *To:* Christian Zunker *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: [ceilometer] radosgw pollster
Hi Christian,
thank you for your feedback and help! Permissions are fine as I tried to poll the Endpoint successfully with curl and the user (key + secret) we created (and is configured in ceilometer.conf).
I saw the requests-aws is used in OSA and it is indeed missing in the kolla container (we use "source" not binary).
https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-...
I will build a new ceilometer container including requests-aws tomorrow to see if this fixes the problem.
All the best,
Florian
*From:* Christian Zunker christian.zunker@codecentric.cloud *Sent:* Wednesday, February 27, 2019 9:09 AM *To:* Engelmann Florian *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: [ceilometer] radosgw pollster Hi Florian,
have you tried different permissions for your ceilometer user in radosgw? According to the docs you need an admin user: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.ht... Our user has these caps: usage=read,write;metadata=read,write;users=read,write;buckets=read,write
We also had to add the requests-aws pip package to query radosgw from ceilometer: https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.htm...
Christian
Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.ch mailto:florian.engelmann@everyware.ch>:
Hi Christian, Am 2/26/19 um 11:00 AM schrieb Christian Zunker: > Hi Florian, > > which version of OpenStack are you using? > The radosgw metric names were different in some versions: > https://bugs.launchpad.net/ceilometer/+bug/1726458 we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. As far as I am able to understand python it looks like the error is happening in polling.manager line 222: https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manager.py#L222 But I do not understand why. I tried to enable debug logging but the error does not log any additional information. The poller is not even trying to reach/poll our RadosGWs. Looks like that manger is blocking those polls. All the best, Florian > > Christian > > Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann > <florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch> <mailto:florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch>>>: > > Hi, > > I failed to poll any usage data from our radosgw. I get > > 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling > pollster radosgw.containers.objects in the context of > radosgw_300s_pollsters > 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent > pollster radosgw.containers.objects from polling [<Project > description=, > domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True, > > [...] > > id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': > u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source > radosgw_300s_pollsters anymore!: PollsterPermanentError > > Configurations like: > cat polling.yaml > --- > sources: > - name: radosgw_300s_pollsters > interval: 300 > meters: > - radosgw.usage > - radosgw.objects > - radosgw.objects.size > - radosgw.objects.containers > - radosgw.containers.objects > - radosgw.containers.objects.size > > > Also tried radosgw.api.requests instead of radowsgw.usage. > > ceilometer.conf > [...] > [service_types] > radosgw = object-store > > [rgw_admin_credentials] > access_key = xxxxx0Z0xxxxxxxxxxxx > secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx > > [rgw_client] > implicit_tenants = true > > Endpoints: > | xxxxxxx | region | swift | object-store | True | admin > | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | > | xxxxxxx | region | swift | object-store | True | > internal > | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | > | xxxxxxx | region | swift | object-store | True | public > | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s | > > Ceilometer user: > { > "user_id": "ceilometer", > "display_name": "ceilometer", > "email": "", > "suspended": 0, > "max_buckets": 1000, > "auid": 0, > "subusers": [], > "keys": [ > { > "user": "ceilometer", > "access_key": "xxxxxxxxxxxxxxxxxx", > "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" > } > ], > "swift_keys": [], > "caps": [ > { > "type": "buckets", > "perm": "read" > }, > { > "type": "metadata", > "perm": "read" > }, > { > "type": "usage", > "perm": "read" > }, > { > "type": "users", > "perm": "read" > } > ], > "op_mask": "read, write, delete", > "default_placement": "", > "placement_tags": [], > "bucket_quota": { > "enabled": false, > "check_on_raw": false, > "max_size": -1, > "max_size_kb": 0, > "max_objects": -1 > }, > "user_quota": { > "enabled": false, > "check_on_raw": false, > "max_size": -1, > "max_size_kb": 0, > "max_objects": -1 > }, > "temp_url_keys": [], > "type": "rgw" > } > > > radosgw config: > [client.rgw.xxxxxxxxxxx] > host = somehost > rgw frontends = "civetweb port=7480 num_threads=512" > rgw num rados handles = 8 > rgw thread pool size = 512 > rgw cache enabled = true > rgw dns name = s3.xxxxxx.xxx > rgw enable usage log = true > rgw usage log tick interval = 30 > rgw realm = public > rgw zonegroup = xxx > rgw zone = xxxxx > rgw resolve cname = False > rgw usage log flush threshold = 1024 > rgw usage max user shards = 1 > rgw usage max shards = 32 > rgw_keystone_url = https://keystone.xxxxxxxxxxxxx > rgw_keystone_admin_domain = default > rgw_keystone_admin_project = service > rgw_keystone_admin_user = swift > rgw_keystone_admin_password = > xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > rgw_keystone_accepted_roles = member,_member_,admin > rgw_keystone_accepted_admin_roles = admin > rgw_keystone_api_version = 3 > rgw_keystone_verify_ssl = false > rgw_keystone_implicit_tenants = true > rgw_keystone_admin_tenant = default > rgw_keystone_revocation_interval = 0 > rgw_keystone_token_cache_size = 0 > rgw_s3_auth_use_keystone = true > rgw_max_attr_size = 1024 > rgw_max_attrs_num_in_req = 32 > rgw_max_attr_name_len = 64 > rgw_swift_account_in_url = true > rgw_swift_versioning_enabled = true > rgw_enable_apis = s3,swift,swift_auth,admin > rgw_swift_enforce_content_length = true > > > > > Any idea whats going on? > > All the best, > Florian > > > -- EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch> web: http://www.everyware.ch
Hi Christian,
all good! I was not getting any result with "openstack resource list" as we got to many resources.
openstack metric resource list --type ceph_account -c id -f value | wc -l
gives results!!!!
Looks like everything is working as expected!
Thank you so much for your help!
All the best, Florian
Am 2/28/19 um 11:21 AM schrieb Florian Engelmann:
Hi Christian,
after adding requests-aws to the container the PollsterPermanentError is gone and the ceilometer polling log looks clean to me:
2019-02-28 10:29:46.163 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.167 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.objects in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.172 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.objects.size in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.177 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.objects.containers in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.182 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.usage in the context of radosgw_300s_pollsters
I still do not get any data in gnocchi:
openstack metric resource list -c type -f value | sort -u generic image instance instance_disk instance_network_interface network volume
My RadosGW logs do show the poller requests: 2019-02-28 10:29:54.767266 7f67a550e700 20 HTTP_ACCEPT=*/* 2019-02-28 10:29:54.767272 7f67a550e700 20 HTTP_ACCEPT_ENCODING=gzip, deflate 2019-02-28 10:29:54.767274 7f67a550e700 20 HTTP_AUTHORIZATION=AWS 0APxxxxxxxxx:vxxxxxxxICq/qQXxxxxxxxxxuuc= [...] 2019-02-28 10:29:54.767294 7f67a550e700 20 REMOTE_ADDR=10.xx.xxx.xxx 2019-02-28 10:29:54.767295 7f67a550e700 20 REQUEST_METHOD=GET 2019-02-28 10:29:54.767296 7f67a550e700 20 REQUEST_URI=/admin/usage 2019-02-28 10:29:54.767297 7f67a550e700 20 SCRIPT_URI=/admin/usage [...] in_hosted_domain_s3website=0 s->info.domain=rgw.xxxxxx s->info.request_uri=/admin/usage 2019-02-28 10:29:54.767479 7f67a550e700 10 handler=16RGWHandler_Usage 2019-02-28 10:29:54.767491 7f67a550e700 2 req 215268:0.000183::GET /admin/usage::getting op 0 2019-02-28 10:29:54.767495 7f67a550e700 10 op=15RGWOp_Usage_Get 2019-02-28 10:29:54.767496 7f67a550e700 2 req 215268:0.000196::GET /admin/usage:get_usage:verifying requester 2019-02-28 10:29:54.767528 7f67a550e700 20 rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2019-02-28 10:29:54.767530 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2019-02-28 10:29:54.767558 7f67a550e700 20 rgw::auth::s3::S3AnonymousEngine denied with reason=-1 2019-02-28 10:29:54.767561 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::AWSv2ExternalAuthStrategy 2019-02-28 10:29:54.767573 7f67a550e700 20 rgw::auth::s3::AWSv2ExternalAuthStrategy: trying rgw::auth::keystone::EC2Engine 2019-02-28 10:29:54.767602 7f67a550e700 10 get_canon_resource(): dest=/admin/usage 2019-02-28 10:29:54.767605 7f67a550e700 10 string_to_sign: 2019-02-28 10:29:54.767630 7f67a550e700 20 sending request to https://keystone.xxxxxxxxx/v3/auth/tokens 2019-02-28 10:29:54.767678 7f67a550e700 20 ssl verification is set to off 2019-02-28 10:29:55.194088 7f67a550e700 20 sending request to https://keystone.xxxxxxxx/v3/s3tokens 2019-02-28 10:29:55.194115 7f67a550e700 20 ssl verification is set to off 2019-02-28 10:29:55.226841 7f67a550e700 20 rgw::auth::keystone::EC2Engine denied with reason=-2028 2019-02-28 10:29:55.226852 7f67a550e700 20 rgw::auth::s3::AWSv2ExternalAuthStrategy denied with reason=-2028 2019-02-28 10:29:55.226855 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine 2019-02-28 10:29:55.226878 7f67a550e700 10 get_canon_resource(): dest=/admin/usage 2019-02-28 10:29:55.226881 7f67a550e700 10 string_to_sign: 2019-02-28 10:29:55.226943 7f67a550e700 15 string_to_sign=GET [...] 2019-02-28 10:29:55.226972 7f67a550e700 15 compare=0 2019-02-28 10:29:55.227099 7f67a550e700 20 rgw::auth::s3::LocalEngine granted access 2019-02-28 10:29:55.227103 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy granted access 2019-02-28 10:29:55.227106 7f67a550e700 2 req 215268:0.459806::GET /admin/usage:get_usage:normalizing buckets and tenants 2019-02-28 10:29:55.227109 7f67a550e700 2 req 215268:0.459809::GET /admin/usage:get_usage:init permissions 2019-02-28 10:29:55.227136 7f67a550e700 2 req 215268:0.459826::GET /admin/usage:get_usage:recalculating target 2019-02-28 10:29:55.227140 7f67a550e700 2 req 215268:0.459841::GET /admin/usage:get_usage:reading permissions 2019-02-28 10:29:55.227142 7f67a550e700 2 req 215268:0.459842::GET /admin/usage:get_usage:init op 2019-02-28 10:29:55.227143 7f67a550e700 2 req 215268:0.459844::GET /admin/usage:get_usage:verifying op mask 2019-02-28 10:29:55.227145 7f67a550e700 20 required_mask= 0 user.op_mask=7 2019-02-28 10:29:55.227146 7f67a550e700 2 req 215268:0.459846::GET /admin/usage:get_usage:verifying op permissions 2019-02-28 10:29:55.227148 7f67a550e700 2 req 215268:0.459848::GET /admin/usage:get_usage:verifying op params 2019-02-28 10:29:55.227149 7f67a550e700 2 req 215268:0.459850::GET /admin/usage:get_usage:pre-executing 2019-02-28 10:29:55.227150 7f67a550e700 2 req 215268:0.459851::GET /admin/usage:get_usage:executing 2019-02-28 10:29:55.232830 7f67a550e700 2 req 215268:0.465530::GET /admin/usage:get_usage:completing 2019-02-28 10:29:55.232858 7f67a550e700 2 req 215268:0.465559::GET /admin/usage:get_usage:op status=0 2019-02-28 10:29:55.232863 7f67a550e700 2 req 215268:0.465564::GET /admin/usage:get_usage:http status=200 2019-02-28 10:29:55.232865 7f67a550e700 1 ====== req done req=0x7f67a55080a0 op status=0 http_status=200 ====== 2019-02-28 10:29:55.232894 7f67a550e700 1 civetweb: 0x55ced7bd2000: 10.0.81.59 - - [28/Feb/2019:10:29:54 +0100] "GET /admin/usage?uid=2534a3e876ee41f088098fxxxxxxxxx%242534a3e876ee41f088098f53xxxxxxx HTTP/1.1" 200 0 - python-requests/2.19.1
Using curl to do the same request gives me the correct usage results: {"entries":[{"user":"a772e4abxxxxxxxx4559d$a772e4ab888exxxxxxxf4559d","buckets":[{"bucket":"","time":"2019-02-26 12:00:00.000000Z","epoch":1551182400,"owner":"a772e4ab88xxxxxxx88e4f039b3430d688f4559d","categories":[{"category":"list_buckets","bytes_sent":36,"bytes_received":0,"ops":3,"successful_ops":0}]},{"bucket":"-","time":"2019-02-17 15:00:00.000000Z","epoch":1550415600,"owner":"a772e4ab88xxxxxxxa772e4ab888e4f039b3430d688f4559d","categories":[{"category":"get_obj","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":0},{"category":"list_bucket","bytes_sent":132,"bytes_received":0,"ops":1,"successful_ops":0}]},{"bucket":"info","time":"2019-02-26 12:00:00.000000Z","epoch":1551182400,"owner":"a772e4ab888e4f039xxxxxx72e4ab888e4f039b3430d688f4559d","categories":[{"category":"RGWMovedPermanently","bytes_sent":0,"bytes_received":0,"ops":3,"successful_ops":3}]},{"bucket":"test","time":"2019-02-17 15:00:00.000000Z","epoch":1550415600,"owner":"a772e4ab8xxxxxxd$a772e4ab888e4f039b3430d688f4559d","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"list_bucket","bytes_sent":1368,"bytes_received":0,"ops":18,"successful_ops":18},{"category":"put_obj","bytes_sent":0,"bytes_received":10,"ops":1,"successful_ops":1}]}]}],"summary":[{"user":"a772e4abxxxx4f0xxx8f4559d$a772e4ab88xxxxxx30d688f4559d","categories":[{"category":"RGWMovedPermanently","bytes_sent":0,"bytes_received":0,"ops":3,"successful_ops":3},{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"get_obj","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":0},{"category":"list_bucket","bytes_sent":1500,"bytes_received":0,"ops":19,"successful_ops":18},{"category":"list_buckets","bytes_sent":36,"bytes_received":0,"ops":3,"successful_ops":0},{"category":"put_obj","bytes_sent":0,"bytes_received":10,"ops":1,"successful_ops":1}],"total":{"bytes_sent":153* Connection #0 to host 10.xxx.xxx.xxx left intact
My custom archive policy (custom_gnocchi_resources.yaml) looks like: [...] - resource_type: ceph_account metrics: radosgw.objects: radosgw.objects.size: radosgw.objects.containers: radosgw.api.request: radosgw.containers.objects: radosgw.containers.objects.size: [...]
and the pipeline: [...] sources: - name: meter_source meters: - "*" sinks: - meter_sink [...] sinks: - name: meter_sink transformers: publishers: - gnocchi://?resources_definition_file=%2Fetc%2Fceilometer%2Fcustom_gnocchi_resources.yaml
[...]
Anything wrong with my archive policy or the pipeline?
All the best, Florian
Am 2/27/19 um 8:38 PM schrieb Engelmann Florian:
Hi Christian,
looks like a hit:
https://github.com/openstack/ceilometer/commit/c9eb2d44df7cafde1294123d66445...
You made my day!
I will test tomorrow and report back!
All the best,
Florian
*From:* Engelmann Florian florian.engelmann@everyware.ch *Sent:* Wednesday, February 27, 2019 8:33 PM *To:* Christian Zunker *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: [ceilometer] radosgw pollster
Hi Christian,
thank you for your feedback and help! Permissions are fine as I tried to poll the Endpoint successfully with curl and the user (key + secret) we created (and is configured in ceilometer.conf).
I saw the requests-aws is used in OSA and it is indeed missing in the kolla container (we use "source" not binary).
https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-...
I will build a new ceilometer container including requests-aws tomorrow to see if this fixes the problem.
All the best,
Florian
*From:* Christian Zunker christian.zunker@codecentric.cloud *Sent:* Wednesday, February 27, 2019 9:09 AM *To:* Engelmann Florian *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: [ceilometer] radosgw pollster Hi Florian,
have you tried different permissions for your ceilometer user in radosgw? According to the docs you need an admin user: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.ht...
Our user has these caps: usage=read,write;metadata=read,write;users=read,write;buckets=read,write
We also had to add the requests-aws pip package to query radosgw from ceilometer: https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.htm...
Christian
Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.ch mailto:florian.engelmann@everyware.ch>:
Hi Christian,
Am 2/26/19 um 11:00 AM schrieb Christian Zunker: > Hi Florian, > > which version of OpenStack are you using? > The radosgw metric names were different in some versions: > https://bugs.launchpad.net/ceilometer/+bug/1726458
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. As far as I am able to understand python it looks like the error is happening in polling.manager line 222:
https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manag...
But I do not understand why. I tried to enable debug logging but the error does not log any additional information. The poller is not even trying to reach/poll our RadosGWs. Looks like that manger is blocking those polls.
All the best, Florian
> > Christian > > Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann > <florian.engelmann@everyware.ch mailto:florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch mailto:florian.engelmann@everyware.ch>>: > > Hi, > > I failed to poll any usage data from our radosgw. I get > > 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling > pollster radosgw.containers.objects in the context of > radosgw_300s_pollsters > 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent > pollster radosgw.containers.objects from polling [<Project > description=, > domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True, > > [...] > > id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': > u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source > radosgw_300s_pollsters anymore!: PollsterPermanentError > > Configurations like: > cat polling.yaml > --- > sources: > - name: radosgw_300s_pollsters > interval: 300 > meters: > - radosgw.usage > - radosgw.objects > - radosgw.objects.size > - radosgw.objects.containers > - radosgw.containers.objects > - radosgw.containers.objects.size > > > Also tried radosgw.api.requests instead of radowsgw.usage. > > ceilometer.conf > [...] > [service_types] > radosgw = object-store > > [rgw_admin_credentials] > access_key = xxxxx0Z0xxxxxxxxxxxx > secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx > > [rgw_client] > implicit_tenants = true > > Endpoints: > | xxxxxxx | region | swift | object-store | True | admin > | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s%C2%A0 | > | xxxxxxx | region | swift | object-store | True | > internal > | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s%C2%A0 | > | xxxxxxx | region | swift | object-store | True | public > | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s | > > Ceilometer user: > { > "user_id": "ceilometer", > "display_name": "ceilometer", > "email": "", > "suspended": 0, > "max_buckets": 1000, > "auid": 0, > "subusers": [], > "keys": [ > { > "user": "ceilometer", > "access_key": "xxxxxxxxxxxxxxxxxx", > "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" > } > ], > "swift_keys": [], > "caps": [ > { > "type": "buckets", > "perm": "read" > }, > { > "type": "metadata", > "perm": "read" > }, > { > "type": "usage", > "perm": "read" > }, > { > "type": "users", > "perm": "read" > } > ], > "op_mask": "read, write, delete", > "default_placement": "", > "placement_tags": [], > "bucket_quota": { > "enabled": false, > "check_on_raw": false, > "max_size": -1, > "max_size_kb": 0, > "max_objects": -1 > }, > "user_quota": { > "enabled": false, > "check_on_raw": false, > "max_size": -1, > "max_size_kb": 0, > "max_objects": -1 > }, > "temp_url_keys": [], > "type": "rgw" > } > > > radosgw config: > [client.rgw.xxxxxxxxxxx] > host = somehost > rgw frontends = "civetweb port=7480 num_threads=512" > rgw num rados handles = 8 > rgw thread pool size = 512 > rgw cache enabled = true > rgw dns name = s3.xxxxxx.xxx > rgw enable usage log = true > rgw usage log tick interval = 30 > rgw realm = public > rgw zonegroup = xxx > rgw zone = xxxxx > rgw resolve cname = False > rgw usage log flush threshold = 1024 > rgw usage max user shards = 1 > rgw usage max shards = 32 > rgw_keystone_url = https://keystone.xxxxxxxxxxxxx > rgw_keystone_admin_domain = default > rgw_keystone_admin_project = service > rgw_keystone_admin_user = swift > rgw_keystone_admin_password = > xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > rgw_keystone_accepted_roles = member,_member_,admin > rgw_keystone_accepted_admin_roles = admin > rgw_keystone_api_version = 3 > rgw_keystone_verify_ssl = false > rgw_keystone_implicit_tenants = true > rgw_keystone_admin_tenant = default > rgw_keystone_revocation_interval = 0 > rgw_keystone_token_cache_size = 0 > rgw_s3_auth_use_keystone = true > rgw_max_attr_size = 1024 > rgw_max_attrs_num_in_req = 32 > rgw_max_attr_name_len = 64 > rgw_swift_account_in_url = true > rgw_swift_versioning_enabled = true > rgw_enable_apis = s3,swift,swift_auth,admin > rgw_swift_enforce_content_length = true > > > > > Any idea whats going on? > > All the best, > Florian > > >
-- EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich
tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.ch mailto:florian.engelmann@everyware.ch web: http://www.everyware.ch
participants (3)
-
Christian Zunker
-
Engelmann Florian
-
Florian Engelmann