[ceilometer] radosgw pollster
Hi, I failed to poll any usage data from our radosgw. I get 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True, [...] id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError Configurations like: cat polling.yaml --- sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size Also tried radosgw.api.requests instead of radowsgw.usage. ceilometer.conf [...] [service_types] radosgw = object-store [rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx [rgw_client] implicit_tenants = true Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | public | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s | Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" } radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true Any idea whats going on? All the best, Florian
Hi Florian, which version of OpenStack are you using? The radosgw metric names were different in some versions: https://bugs.launchpad.net/ceilometer/+bug/1726458 Christian Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann < florian.engelmann@everyware.ch>:
Hi,
I failed to poll any usage data from our radosgw. I get
2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True,
[...]
id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError
Configurations like: cat polling.yaml --- sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size
Also tried radosgw.api.requests instead of radowsgw.usage.
ceilometer.conf [...] [service_types] radosgw = object-store
[rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx
[rgw_client] implicit_tenants = true
Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | public | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s |
Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" }
radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true
Any idea whats going on?
All the best, Florian
Hi Christian, Am 2/26/19 um 11:00 AM schrieb Christian Zunker:
Hi Florian,
which version of OpenStack are you using? The radosgw metric names were different in some versions: https://bugs.launchpad.net/ceilometer/+bug/1726458
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. As far as I am able to understand python it looks like the error is happening in polling.manager line 222: https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manag... But I do not understand why. I tried to enable debug logging but the error does not log any additional information. The poller is not even trying to reach/poll our RadosGWs. Looks like that manger is blocking those polls. All the best, Florian
Christian
Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch>>:
Hi,
I failed to poll any usage data from our radosgw. I get
2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True,
[...]
id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError
Configurations like: cat polling.yaml --- sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size
Also tried radosgw.api.requests instead of radowsgw.usage.
ceilometer.conf [...] [service_types] radosgw = object-store
[rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx
[rgw_client] implicit_tenants = true
Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | public | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s |
Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" }
radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true
Any idea whats going on?
All the best, Florian
-- EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.ch web: http://www.everyware.ch
Hi Florian, have you tried different permissions for your ceilometer user in radosgw? According to the docs you need an admin user: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.ht... Our user has these caps: usage=read,write;metadata=read,write;users=read,write;buckets=read,write We also had to add the requests-aws pip package to query radosgw from ceilometer: https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.htm... Christian Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann < florian.engelmann@everyware.ch>:
Hi Christian,
Am 2/26/19 um 11:00 AM schrieb Christian Zunker:
Hi Florian,
which version of OpenStack are you using? The radosgw metric names were different in some versions: https://bugs.launchpad.net/ceilometer/+bug/1726458
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. As far as I am able to understand python it looks like the error is happening in polling.manager line 222:
https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manag...
But I do not understand why. I tried to enable debug logging but the error does not log any additional information. The poller is not even trying to reach/poll our RadosGWs. Looks like that manger is blocking those polls.
All the best, Florian
Christian
Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch
:
Hi,
I failed to poll any usage data from our radosgw. I get
2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-]
Polling
pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-]
Prevent
pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True,
[...]
id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError
Configurations like: cat polling.yaml --- sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size
Also tried radosgw.api.requests instead of radowsgw.usage.
ceilometer.conf [...] [service_types] radosgw = object-store
[rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx
[rgw_client] implicit_tenants = true
Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True |
public
| https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s |
Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" }
radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true
Any idea whats going on?
All the best, Florian
--
EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich
tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.ch web: http://www.everyware.ch
Hi Christian, thank you for your feedback and help! Permissions are fine as I tried to poll the Endpoint successfully with curl and the user (key + secret) we created (and is configured in ceilometer.conf). I saw the requests-aws is used in OSA and it is indeed missing in the kolla container (we use "source" not binary). https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-... I will build a new ceilometer container including requests-aws tomorrow to see if this fixes the problem. All the best, Florian ________________________________ From: Christian Zunker <christian.zunker@codecentric.cloud> Sent: Wednesday, February 27, 2019 9:09 AM To: Engelmann Florian Cc: openstack-discuss@lists.openstack.org Subject: Re: [ceilometer] radosgw pollster Hi Florian, have you tried different permissions for your ceilometer user in radosgw? According to the docs you need an admin user: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.ht... Our user has these caps: usage=read,write;metadata=read,write;users=read,write;buckets=read,write We also had to add the requests-aws pip package to query radosgw from ceilometer: https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.htm... Christian Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.ch<mailto:florian.engelmann@everyware.ch>>: Hi Christian, Am 2/26/19 um 11:00 AM schrieb Christian Zunker:
Hi Florian,
which version of OpenStack are you using? The radosgw metric names were different in some versions: https://bugs.launchpad.net/ceilometer/+bug/1726458
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. As far as I am able to understand python it looks like the error is happening in polling.manager line 222: https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manag... But I do not understand why. I tried to enable debug logging but the error does not log any additional information. The poller is not even trying to reach/poll our RadosGWs. Looks like that manger is blocking those polls. All the best, Florian
Christian
Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.ch<mailto:florian.engelmann@everyware.ch> <mailto:florian.engelmann@everyware.ch<mailto:florian.engelmann@everyware.ch>>>:
Hi,
I failed to poll any usage data from our radosgw. I get
2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True,
[...]
id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError
Configurations like: cat polling.yaml --- sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size
Also tried radosgw.api.requests instead of radowsgw.usage.
ceilometer.conf [...] [service_types] radosgw = object-store
[rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx
[rgw_client] implicit_tenants = true
Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | public | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s |
Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" }
radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true
Any idea whats going on?
All the best, Florian
-- EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.ch<mailto:florian.engelmann@everyware.ch> web: http://www.everyware.ch
Hi Christian, looks like a hit: https://github.com/openstack/ceilometer/commit/c9eb2d44df7cafde1294123d66445... You made my day! I will test tomorrow and report back! All the best, Florian ________________________________ From: Engelmann Florian <florian.engelmann@everyware.ch> Sent: Wednesday, February 27, 2019 8:33 PM To: Christian Zunker Cc: openstack-discuss@lists.openstack.org Subject: Re: [ceilometer] radosgw pollster Hi Christian, thank you for your feedback and help! Permissions are fine as I tried to poll the Endpoint successfully with curl and the user (key + secret) we created (and is configured in ceilometer.conf). I saw the requests-aws is used in OSA and it is indeed missing in the kolla container (we use "source" not binary). https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-... I will build a new ceilometer container including requests-aws tomorrow to see if this fixes the problem. All the best, Florian ________________________________ From: Christian Zunker <christian.zunker@codecentric.cloud> Sent: Wednesday, February 27, 2019 9:09 AM To: Engelmann Florian Cc: openstack-discuss@lists.openstack.org Subject: Re: [ceilometer] radosgw pollster Hi Florian, have you tried different permissions for your ceilometer user in radosgw? According to the docs you need an admin user: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.ht... Our user has these caps: usage=read,write;metadata=read,write;users=read,write;buckets=read,write We also had to add the requests-aws pip package to query radosgw from ceilometer: https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.htm... Christian Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.ch<mailto:florian.engelmann@everyware.ch>>: Hi Christian, Am 2/26/19 um 11:00 AM schrieb Christian Zunker:
Hi Florian,
which version of OpenStack are you using? The radosgw metric names were different in some versions: https://bugs.launchpad.net/ceilometer/+bug/1726458
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. As far as I am able to understand python it looks like the error is happening in polling.manager line 222: https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manag... But I do not understand why. I tried to enable debug logging but the error does not log any additional information. The poller is not even trying to reach/poll our RadosGWs. Looks like that manger is blocking those polls. All the best, Florian
Christian
Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.ch<mailto:florian.engelmann@everyware.ch> <mailto:florian.engelmann@everyware.ch<mailto:florian.engelmann@everyware.ch>>>:
Hi,
I failed to poll any usage data from our radosgw. I get
2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent pollster radosgw.containers.objects from polling [<Project description=, domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True,
[...]
id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source radosgw_300s_pollsters anymore!: PollsterPermanentError
Configurations like: cat polling.yaml --- sources: - name: radosgw_300s_pollsters interval: 300 meters: - radosgw.usage - radosgw.objects - radosgw.objects.size - radosgw.objects.containers - radosgw.containers.objects - radosgw.containers.objects.size
Also tried radosgw.api.requests instead of radowsgw.usage.
ceilometer.conf [...] [service_types] radosgw = object-store
[rgw_admin_credentials] access_key = xxxxx0Z0xxxxxxxxxxxx secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx
[rgw_client] implicit_tenants = true
Endpoints: | xxxxxxx | region | swift | object-store | True | admin | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | internal | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | | xxxxxxx | region | swift | object-store | True | public | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s |
Ceilometer user: { "user_id": "ceilometer", "display_name": "ceilometer", "email": "", "suspended": 0, "max_buckets": 1000, "auid": 0, "subusers": [], "keys": [ { "user": "ceilometer", "access_key": "xxxxxxxxxxxxxxxxxx", "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "read" }, { "type": "metadata", "perm": "read" }, { "type": "usage", "perm": "read" }, { "type": "users", "perm": "read" } ], "op_mask": "read, write, delete", "default_placement": "", "placement_tags": [], "bucket_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "user_quota": { "enabled": false, "check_on_raw": false, "max_size": -1, "max_size_kb": 0, "max_objects": -1 }, "temp_url_keys": [], "type": "rgw" }
radosgw config: [client.rgw.xxxxxxxxxxx] host = somehost rgw frontends = "civetweb port=7480 num_threads=512" rgw num rados handles = 8 rgw thread pool size = 512 rgw cache enabled = true rgw dns name = s3.xxxxxx.xxx rgw enable usage log = true rgw usage log tick interval = 30 rgw realm = public rgw zonegroup = xxx rgw zone = xxxxx rgw resolve cname = False rgw usage log flush threshold = 1024 rgw usage max user shards = 1 rgw usage max shards = 32 rgw_keystone_url = https://keystone.xxxxxxxxxxxxx rgw_keystone_admin_domain = default rgw_keystone_admin_project = service rgw_keystone_admin_user = swift rgw_keystone_admin_password = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx rgw_keystone_accepted_roles = member,_member_,admin rgw_keystone_accepted_admin_roles = admin rgw_keystone_api_version = 3 rgw_keystone_verify_ssl = false rgw_keystone_implicit_tenants = true rgw_keystone_admin_tenant = default rgw_keystone_revocation_interval = 0 rgw_keystone_token_cache_size = 0 rgw_s3_auth_use_keystone = true rgw_max_attr_size = 1024 rgw_max_attrs_num_in_req = 32 rgw_max_attr_name_len = 64 rgw_swift_account_in_url = true rgw_swift_versioning_enabled = true rgw_enable_apis = s3,swift,swift_auth,admin rgw_swift_enforce_content_length = true
Any idea whats going on?
All the best, Florian
-- EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.ch<mailto:florian.engelmann@everyware.ch> web: http://www.everyware.ch
Hi Christian, after adding requests-aws to the container the PollsterPermanentError is gone and the ceilometer polling log looks clean to me: 2019-02-28 10:29:46.163 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.167 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.objects in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.172 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.objects.size in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.177 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.objects.containers in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.182 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.usage in the context of radosgw_300s_pollsters I still do not get any data in gnocchi: openstack metric resource list -c type -f value | sort -u generic image instance instance_disk instance_network_interface network volume My RadosGW logs do show the poller requests: 2019-02-28 10:29:54.767266 7f67a550e700 20 HTTP_ACCEPT=*/* 2019-02-28 10:29:54.767272 7f67a550e700 20 HTTP_ACCEPT_ENCODING=gzip, deflate 2019-02-28 10:29:54.767274 7f67a550e700 20 HTTP_AUTHORIZATION=AWS 0APxxxxxxxxx:vxxxxxxxICq/qQXxxxxxxxxxuuc= [...] 2019-02-28 10:29:54.767294 7f67a550e700 20 REMOTE_ADDR=10.xx.xxx.xxx 2019-02-28 10:29:54.767295 7f67a550e700 20 REQUEST_METHOD=GET 2019-02-28 10:29:54.767296 7f67a550e700 20 REQUEST_URI=/admin/usage 2019-02-28 10:29:54.767297 7f67a550e700 20 SCRIPT_URI=/admin/usage [...] in_hosted_domain_s3website=0 s->info.domain=rgw.xxxxxx s->info.request_uri=/admin/usage 2019-02-28 10:29:54.767479 7f67a550e700 10 handler=16RGWHandler_Usage 2019-02-28 10:29:54.767491 7f67a550e700 2 req 215268:0.000183::GET /admin/usage::getting op 0 2019-02-28 10:29:54.767495 7f67a550e700 10 op=15RGWOp_Usage_Get 2019-02-28 10:29:54.767496 7f67a550e700 2 req 215268:0.000196::GET /admin/usage:get_usage:verifying requester 2019-02-28 10:29:54.767528 7f67a550e700 20 rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2019-02-28 10:29:54.767530 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2019-02-28 10:29:54.767558 7f67a550e700 20 rgw::auth::s3::S3AnonymousEngine denied with reason=-1 2019-02-28 10:29:54.767561 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::AWSv2ExternalAuthStrategy 2019-02-28 10:29:54.767573 7f67a550e700 20 rgw::auth::s3::AWSv2ExternalAuthStrategy: trying rgw::auth::keystone::EC2Engine 2019-02-28 10:29:54.767602 7f67a550e700 10 get_canon_resource(): dest=/admin/usage 2019-02-28 10:29:54.767605 7f67a550e700 10 string_to_sign: 2019-02-28 10:29:54.767630 7f67a550e700 20 sending request to https://keystone.xxxxxxxxx/v3/auth/tokens 2019-02-28 10:29:54.767678 7f67a550e700 20 ssl verification is set to off 2019-02-28 10:29:55.194088 7f67a550e700 20 sending request to https://keystone.xxxxxxxx/v3/s3tokens 2019-02-28 10:29:55.194115 7f67a550e700 20 ssl verification is set to off 2019-02-28 10:29:55.226841 7f67a550e700 20 rgw::auth::keystone::EC2Engine denied with reason=-2028 2019-02-28 10:29:55.226852 7f67a550e700 20 rgw::auth::s3::AWSv2ExternalAuthStrategy denied with reason=-2028 2019-02-28 10:29:55.226855 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine 2019-02-28 10:29:55.226878 7f67a550e700 10 get_canon_resource(): dest=/admin/usage 2019-02-28 10:29:55.226881 7f67a550e700 10 string_to_sign: 2019-02-28 10:29:55.226943 7f67a550e700 15 string_to_sign=GET [...] 2019-02-28 10:29:55.226972 7f67a550e700 15 compare=0 2019-02-28 10:29:55.227099 7f67a550e700 20 rgw::auth::s3::LocalEngine granted access 2019-02-28 10:29:55.227103 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy granted access 2019-02-28 10:29:55.227106 7f67a550e700 2 req 215268:0.459806::GET /admin/usage:get_usage:normalizing buckets and tenants 2019-02-28 10:29:55.227109 7f67a550e700 2 req 215268:0.459809::GET /admin/usage:get_usage:init permissions 2019-02-28 10:29:55.227136 7f67a550e700 2 req 215268:0.459826::GET /admin/usage:get_usage:recalculating target 2019-02-28 10:29:55.227140 7f67a550e700 2 req 215268:0.459841::GET /admin/usage:get_usage:reading permissions 2019-02-28 10:29:55.227142 7f67a550e700 2 req 215268:0.459842::GET /admin/usage:get_usage:init op 2019-02-28 10:29:55.227143 7f67a550e700 2 req 215268:0.459844::GET /admin/usage:get_usage:verifying op mask 2019-02-28 10:29:55.227145 7f67a550e700 20 required_mask= 0 user.op_mask=7 2019-02-28 10:29:55.227146 7f67a550e700 2 req 215268:0.459846::GET /admin/usage:get_usage:verifying op permissions 2019-02-28 10:29:55.227148 7f67a550e700 2 req 215268:0.459848::GET /admin/usage:get_usage:verifying op params 2019-02-28 10:29:55.227149 7f67a550e700 2 req 215268:0.459850::GET /admin/usage:get_usage:pre-executing 2019-02-28 10:29:55.227150 7f67a550e700 2 req 215268:0.459851::GET /admin/usage:get_usage:executing 2019-02-28 10:29:55.232830 7f67a550e700 2 req 215268:0.465530::GET /admin/usage:get_usage:completing 2019-02-28 10:29:55.232858 7f67a550e700 2 req 215268:0.465559::GET /admin/usage:get_usage:op status=0 2019-02-28 10:29:55.232863 7f67a550e700 2 req 215268:0.465564::GET /admin/usage:get_usage:http status=200 2019-02-28 10:29:55.232865 7f67a550e700 1 ====== req done req=0x7f67a55080a0 op status=0 http_status=200 ====== 2019-02-28 10:29:55.232894 7f67a550e700 1 civetweb: 0x55ced7bd2000: 10.0.81.59 - - [28/Feb/2019:10:29:54 +0100] "GET /admin/usage?uid=2534a3e876ee41f088098fxxxxxxxxx%242534a3e876ee41f088098f53xxxxxxx HTTP/1.1" 200 0 - python-requests/2.19.1 Using curl to do the same request gives me the correct usage results: {"entries":[{"user":"a772e4abxxxxxxxx4559d$a772e4ab888exxxxxxxf4559d","buckets":[{"bucket":"","time":"2019-02-26 12:00:00.000000Z","epoch":1551182400,"owner":"a772e4ab88xxxxxxx88e4f039b3430d688f4559d","categories":[{"category":"list_buckets","bytes_sent":36,"bytes_received":0,"ops":3,"successful_ops":0}]},{"bucket":"-","time":"2019-02-17 15:00:00.000000Z","epoch":1550415600,"owner":"a772e4ab88xxxxxxxa772e4ab888e4f039b3430d688f4559d","categories":[{"category":"get_obj","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":0},{"category":"list_bucket","bytes_sent":132,"bytes_received":0,"ops":1,"successful_ops":0}]},{"bucket":"info","time":"2019-02-26 12:00:00.000000Z","epoch":1551182400,"owner":"a772e4ab888e4f039xxxxxx72e4ab888e4f039b3430d688f4559d","categories":[{"category":"RGWMovedPermanently","bytes_sent":0,"bytes_received":0,"ops":3,"successful_ops":3}]},{"bucket":"test","time":"2019-02-17 15:00:00.000000Z","epoch":1550415600,"owner":"a772e4ab8xxxxxxd$a772e4ab888e4f039b3430d688f4559d","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"list_bucket","bytes_sent":1368,"bytes_received":0,"ops":18,"successful_ops":18},{"category":"put_obj","bytes_sent":0,"bytes_received":10,"ops":1,"successful_ops":1}]}]}],"summary":[{"user":"a772e4abxxxx4f0xxx8f4559d$a772e4ab88xxxxxx30d688f4559d","categories":[{"category":"RGWMovedPermanently","bytes_sent":0,"bytes_received":0,"ops":3,"successful_ops":3},{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"get_obj","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":0},{"category":"list_bucket","bytes_sent":1500,"bytes_received":0,"ops":19,"successful_ops":18},{"category":"list_buckets","bytes_sent":36,"bytes_received":0,"ops":3,"successful_ops":0},{"category":"put_obj","bytes_sent":0,"bytes_received":10,"ops":1,"successful_ops":1}],"total":{"bytes_sent":153* Connection #0 to host 10.xxx.xxx.xxx left intact My custom archive policy (custom_gnocchi_resources.yaml) looks like: [...] - resource_type: ceph_account metrics: radosgw.objects: radosgw.objects.size: radosgw.objects.containers: radosgw.api.request: radosgw.containers.objects: radosgw.containers.objects.size: [...] and the pipeline: [...] sources: - name: meter_source meters: - "*" sinks: - meter_sink [...] sinks: - name: meter_sink transformers: publishers: - gnocchi://?resources_definition_file=%2Fetc%2Fceilometer%2Fcustom_gnocchi_resources.yaml [...] Anything wrong with my archive policy or the pipeline? All the best, Florian Am 2/27/19 um 8:38 PM schrieb Engelmann Florian:
Hi Christian,
looks like a hit:
https://github.com/openstack/ceilometer/commit/c9eb2d44df7cafde1294123d66445...
You made my day!
I will test tomorrow and report back!
All the best,
Florian
------------------------------------------------------------------------ *From:* Engelmann Florian <florian.engelmann@everyware.ch> *Sent:* Wednesday, February 27, 2019 8:33 PM *To:* Christian Zunker *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: [ceilometer] radosgw pollster
Hi Christian,
thank you for your feedback and help! Permissions are fine as I tried to poll the Endpoint successfully with curl and the user (key + secret) we created (and is configured in ceilometer.conf).
I saw the requests-aws is used in OSA and it is indeed missing in the kolla container (we use "source" not binary).
https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-...
I will build a new ceilometer container including requests-aws tomorrow to see if this fixes the problem.
All the best,
Florian
------------------------------------------------------------------------ *From:* Christian Zunker <christian.zunker@codecentric.cloud> *Sent:* Wednesday, February 27, 2019 9:09 AM *To:* Engelmann Florian *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: [ceilometer] radosgw pollster Hi Florian,
have you tried different permissions for your ceilometer user in radosgw? According to the docs you need an admin user: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.ht... Our user has these caps: usage=read,write;metadata=read,write;users=read,write;buckets=read,write
We also had to add the requests-aws pip package to query radosgw from ceilometer: https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.htm...
Christian
Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch>>:
Hi Christian,
Am 2/26/19 um 11:00 AM schrieb Christian Zunker: > Hi Florian, > > which version of OpenStack are you using? > The radosgw metric names were different in some versions: > https://bugs.launchpad.net/ceilometer/+bug/1726458
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. As far as I am able to understand python it looks like the error is happening in polling.manager line 222:
https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manag...
But I do not understand why. I tried to enable debug logging but the error does not log any additional information. The poller is not even trying to reach/poll our RadosGWs. Looks like that manger is blocking those polls.
All the best, Florian
> > Christian > > Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann > <florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch> <mailto:florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch>>>: > > Hi, > > I failed to poll any usage data from our radosgw. I get > > 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling > pollster radosgw.containers.objects in the context of > radosgw_300s_pollsters > 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent > pollster radosgw.containers.objects from polling [<Project > description=, > domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True, > > [...] > > id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': > u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source > radosgw_300s_pollsters anymore!: PollsterPermanentError > > Configurations like: > cat polling.yaml > --- > sources: > - name: radosgw_300s_pollsters > interval: 300 > meters: > - radosgw.usage > - radosgw.objects > - radosgw.objects.size > - radosgw.objects.containers > - radosgw.containers.objects > - radosgw.containers.objects.size > > > Also tried radosgw.api.requests instead of radowsgw.usage. > > ceilometer.conf > [...] > [service_types] > radosgw = object-store > > [rgw_admin_credentials] > access_key = xxxxx0Z0xxxxxxxxxxxx > secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx > > [rgw_client] > implicit_tenants = true > > Endpoints: > | xxxxxxx | region | swift | object-store | True | admin > | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | > | xxxxxxx | region | swift | object-store | True | > internal > | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | > | xxxxxxx | region | swift | object-store | True | public > | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s | > > Ceilometer user: > { > "user_id": "ceilometer", > "display_name": "ceilometer", > "email": "", > "suspended": 0, > "max_buckets": 1000, > "auid": 0, > "subusers": [], > "keys": [ > { > "user": "ceilometer", > "access_key": "xxxxxxxxxxxxxxxxxx", > "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" > } > ], > "swift_keys": [], > "caps": [ > { > "type": "buckets", > "perm": "read" > }, > { > "type": "metadata", > "perm": "read" > }, > { > "type": "usage", > "perm": "read" > }, > { > "type": "users", > "perm": "read" > } > ], > "op_mask": "read, write, delete", > "default_placement": "", > "placement_tags": [], > "bucket_quota": { > "enabled": false, > "check_on_raw": false, > "max_size": -1, > "max_size_kb": 0, > "max_objects": -1 > }, > "user_quota": { > "enabled": false, > "check_on_raw": false, > "max_size": -1, > "max_size_kb": 0, > "max_objects": -1 > }, > "temp_url_keys": [], > "type": "rgw" > } > > > radosgw config: > [client.rgw.xxxxxxxxxxx] > host = somehost > rgw frontends = "civetweb port=7480 num_threads=512" > rgw num rados handles = 8 > rgw thread pool size = 512 > rgw cache enabled = true > rgw dns name = s3.xxxxxx.xxx > rgw enable usage log = true > rgw usage log tick interval = 30 > rgw realm = public > rgw zonegroup = xxx > rgw zone = xxxxx > rgw resolve cname = False > rgw usage log flush threshold = 1024 > rgw usage max user shards = 1 > rgw usage max shards = 32 > rgw_keystone_url = https://keystone.xxxxxxxxxxxxx > rgw_keystone_admin_domain = default > rgw_keystone_admin_project = service > rgw_keystone_admin_user = swift > rgw_keystone_admin_password = > xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > rgw_keystone_accepted_roles = member,_member_,admin > rgw_keystone_accepted_admin_roles = admin > rgw_keystone_api_version = 3 > rgw_keystone_verify_ssl = false > rgw_keystone_implicit_tenants = true > rgw_keystone_admin_tenant = default > rgw_keystone_revocation_interval = 0 > rgw_keystone_token_cache_size = 0 > rgw_s3_auth_use_keystone = true > rgw_max_attr_size = 1024 > rgw_max_attrs_num_in_req = 32 > rgw_max_attr_name_len = 64 > rgw_swift_account_in_url = true > rgw_swift_versioning_enabled = true > rgw_enable_apis = s3,swift,swift_auth,admin > rgw_swift_enforce_content_length = true > > > > > Any idea whats going on? > > All the best, > Florian > > >
--
EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich
tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch> web: http://www.everyware.ch
-- EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.ch web: http://www.everyware.ch
Hi Christian, all good! I was not getting any result with "openstack resource list" as we got to many resources. openstack metric resource list --type ceph_account -c id -f value | wc -l gives results!!!! Looks like everything is working as expected! Thank you so much for your help! All the best, Florian Am 2/28/19 um 11:21 AM schrieb Florian Engelmann:
Hi Christian,
after adding requests-aws to the container the PollsterPermanentError is gone and the ceilometer polling log looks clean to me:
2019-02-28 10:29:46.163 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.containers.objects in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.167 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.objects in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.172 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.objects.size in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.177 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.objects.containers in the context of radosgw_300s_pollsters 2019-02-28 10:29:46.182 24 INFO ceilometer.polling.manager [-] Polling pollster radosgw.usage in the context of radosgw_300s_pollsters
I still do not get any data in gnocchi:
openstack metric resource list -c type -f value | sort -u generic image instance instance_disk instance_network_interface network volume
My RadosGW logs do show the poller requests: 2019-02-28 10:29:54.767266 7f67a550e700 20 HTTP_ACCEPT=*/* 2019-02-28 10:29:54.767272 7f67a550e700 20 HTTP_ACCEPT_ENCODING=gzip, deflate 2019-02-28 10:29:54.767274 7f67a550e700 20 HTTP_AUTHORIZATION=AWS 0APxxxxxxxxx:vxxxxxxxICq/qQXxxxxxxxxxuuc= [...] 2019-02-28 10:29:54.767294 7f67a550e700 20 REMOTE_ADDR=10.xx.xxx.xxx 2019-02-28 10:29:54.767295 7f67a550e700 20 REQUEST_METHOD=GET 2019-02-28 10:29:54.767296 7f67a550e700 20 REQUEST_URI=/admin/usage 2019-02-28 10:29:54.767297 7f67a550e700 20 SCRIPT_URI=/admin/usage [...] in_hosted_domain_s3website=0 s->info.domain=rgw.xxxxxx s->info.request_uri=/admin/usage 2019-02-28 10:29:54.767479 7f67a550e700 10 handler=16RGWHandler_Usage 2019-02-28 10:29:54.767491 7f67a550e700 2 req 215268:0.000183::GET /admin/usage::getting op 0 2019-02-28 10:29:54.767495 7f67a550e700 10 op=15RGWOp_Usage_Get 2019-02-28 10:29:54.767496 7f67a550e700 2 req 215268:0.000196::GET /admin/usage:get_usage:verifying requester 2019-02-28 10:29:54.767528 7f67a550e700 20 rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy 2019-02-28 10:29:54.767530 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine 2019-02-28 10:29:54.767558 7f67a550e700 20 rgw::auth::s3::S3AnonymousEngine denied with reason=-1 2019-02-28 10:29:54.767561 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::AWSv2ExternalAuthStrategy 2019-02-28 10:29:54.767573 7f67a550e700 20 rgw::auth::s3::AWSv2ExternalAuthStrategy: trying rgw::auth::keystone::EC2Engine 2019-02-28 10:29:54.767602 7f67a550e700 10 get_canon_resource(): dest=/admin/usage 2019-02-28 10:29:54.767605 7f67a550e700 10 string_to_sign: 2019-02-28 10:29:54.767630 7f67a550e700 20 sending request to https://keystone.xxxxxxxxx/v3/auth/tokens 2019-02-28 10:29:54.767678 7f67a550e700 20 ssl verification is set to off 2019-02-28 10:29:55.194088 7f67a550e700 20 sending request to https://keystone.xxxxxxxx/v3/s3tokens 2019-02-28 10:29:55.194115 7f67a550e700 20 ssl verification is set to off 2019-02-28 10:29:55.226841 7f67a550e700 20 rgw::auth::keystone::EC2Engine denied with reason=-2028 2019-02-28 10:29:55.226852 7f67a550e700 20 rgw::auth::s3::AWSv2ExternalAuthStrategy denied with reason=-2028 2019-02-28 10:29:55.226855 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine 2019-02-28 10:29:55.226878 7f67a550e700 10 get_canon_resource(): dest=/admin/usage 2019-02-28 10:29:55.226881 7f67a550e700 10 string_to_sign: 2019-02-28 10:29:55.226943 7f67a550e700 15 string_to_sign=GET [...] 2019-02-28 10:29:55.226972 7f67a550e700 15 compare=0 2019-02-28 10:29:55.227099 7f67a550e700 20 rgw::auth::s3::LocalEngine granted access 2019-02-28 10:29:55.227103 7f67a550e700 20 rgw::auth::s3::AWSAuthStrategy granted access 2019-02-28 10:29:55.227106 7f67a550e700 2 req 215268:0.459806::GET /admin/usage:get_usage:normalizing buckets and tenants 2019-02-28 10:29:55.227109 7f67a550e700 2 req 215268:0.459809::GET /admin/usage:get_usage:init permissions 2019-02-28 10:29:55.227136 7f67a550e700 2 req 215268:0.459826::GET /admin/usage:get_usage:recalculating target 2019-02-28 10:29:55.227140 7f67a550e700 2 req 215268:0.459841::GET /admin/usage:get_usage:reading permissions 2019-02-28 10:29:55.227142 7f67a550e700 2 req 215268:0.459842::GET /admin/usage:get_usage:init op 2019-02-28 10:29:55.227143 7f67a550e700 2 req 215268:0.459844::GET /admin/usage:get_usage:verifying op mask 2019-02-28 10:29:55.227145 7f67a550e700 20 required_mask= 0 user.op_mask=7 2019-02-28 10:29:55.227146 7f67a550e700 2 req 215268:0.459846::GET /admin/usage:get_usage:verifying op permissions 2019-02-28 10:29:55.227148 7f67a550e700 2 req 215268:0.459848::GET /admin/usage:get_usage:verifying op params 2019-02-28 10:29:55.227149 7f67a550e700 2 req 215268:0.459850::GET /admin/usage:get_usage:pre-executing 2019-02-28 10:29:55.227150 7f67a550e700 2 req 215268:0.459851::GET /admin/usage:get_usage:executing 2019-02-28 10:29:55.232830 7f67a550e700 2 req 215268:0.465530::GET /admin/usage:get_usage:completing 2019-02-28 10:29:55.232858 7f67a550e700 2 req 215268:0.465559::GET /admin/usage:get_usage:op status=0 2019-02-28 10:29:55.232863 7f67a550e700 2 req 215268:0.465564::GET /admin/usage:get_usage:http status=200 2019-02-28 10:29:55.232865 7f67a550e700 1 ====== req done req=0x7f67a55080a0 op status=0 http_status=200 ====== 2019-02-28 10:29:55.232894 7f67a550e700 1 civetweb: 0x55ced7bd2000: 10.0.81.59 - - [28/Feb/2019:10:29:54 +0100] "GET /admin/usage?uid=2534a3e876ee41f088098fxxxxxxxxx%242534a3e876ee41f088098f53xxxxxxx HTTP/1.1" 200 0 - python-requests/2.19.1
Using curl to do the same request gives me the correct usage results: {"entries":[{"user":"a772e4abxxxxxxxx4559d$a772e4ab888exxxxxxxf4559d","buckets":[{"bucket":"","time":"2019-02-26 12:00:00.000000Z","epoch":1551182400,"owner":"a772e4ab88xxxxxxx88e4f039b3430d688f4559d","categories":[{"category":"list_buckets","bytes_sent":36,"bytes_received":0,"ops":3,"successful_ops":0}]},{"bucket":"-","time":"2019-02-17 15:00:00.000000Z","epoch":1550415600,"owner":"a772e4ab88xxxxxxxa772e4ab888e4f039b3430d688f4559d","categories":[{"category":"get_obj","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":0},{"category":"list_bucket","bytes_sent":132,"bytes_received":0,"ops":1,"successful_ops":0}]},{"bucket":"info","time":"2019-02-26 12:00:00.000000Z","epoch":1551182400,"owner":"a772e4ab888e4f039xxxxxx72e4ab888e4f039b3430d688f4559d","categories":[{"category":"RGWMovedPermanently","bytes_sent":0,"bytes_received":0,"ops":3,"successful_ops":3}]},{"bucket":"test","time":"2019-02-17 15:00:00.000000Z","epoch":1550415600,"owner":"a772e4ab8xxxxxxd$a772e4ab888e4f039b3430d688f4559d","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"list_bucket","bytes_sent":1368,"bytes_received":0,"ops":18,"successful_ops":18},{"category":"put_obj","bytes_sent":0,"bytes_received":10,"ops":1,"successful_ops":1}]}]}],"summary":[{"user":"a772e4abxxxx4f0xxx8f4559d$a772e4ab88xxxxxx30d688f4559d","categories":[{"category":"RGWMovedPermanently","bytes_sent":0,"bytes_received":0,"ops":3,"successful_ops":3},{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"get_obj","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":0},{"category":"list_bucket","bytes_sent":1500,"bytes_received":0,"ops":19,"successful_ops":18},{"category":"list_buckets","bytes_sent":36,"bytes_received":0,"ops":3,"successful_ops":0},{"category":"put_obj","bytes_sent":0,"bytes_received":10,"ops":1,"successful_ops":1}],"total":{"bytes_sent":153* Connection #0 to host 10.xxx.xxx.xxx left intact
My custom archive policy (custom_gnocchi_resources.yaml) looks like: [...] - resource_type: ceph_account metrics: radosgw.objects: radosgw.objects.size: radosgw.objects.containers: radosgw.api.request: radosgw.containers.objects: radosgw.containers.objects.size: [...]
and the pipeline: [...] sources: - name: meter_source meters: - "*" sinks: - meter_sink [...] sinks: - name: meter_sink transformers: publishers: - gnocchi://?resources_definition_file=%2Fetc%2Fceilometer%2Fcustom_gnocchi_resources.yaml
[...]
Anything wrong with my archive policy or the pipeline?
All the best, Florian
Am 2/27/19 um 8:38 PM schrieb Engelmann Florian:
Hi Christian,
looks like a hit:
https://github.com/openstack/ceilometer/commit/c9eb2d44df7cafde1294123d66445...
You made my day!
I will test tomorrow and report back!
All the best,
Florian
------------------------------------------------------------------------ *From:* Engelmann Florian <florian.engelmann@everyware.ch> *Sent:* Wednesday, February 27, 2019 8:33 PM *To:* Christian Zunker *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: [ceilometer] radosgw pollster
Hi Christian,
thank you for your feedback and help! Permissions are fine as I tried to poll the Endpoint successfully with curl and the user (key + secret) we created (and is configured in ceilometer.conf).
I saw the requests-aws is used in OSA and it is indeed missing in the kolla container (we use "source" not binary).
https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-...
I will build a new ceilometer container including requests-aws tomorrow to see if this fixes the problem.
All the best,
Florian
------------------------------------------------------------------------ *From:* Christian Zunker <christian.zunker@codecentric.cloud> *Sent:* Wednesday, February 27, 2019 9:09 AM *To:* Engelmann Florian *Cc:* openstack-discuss@lists.openstack.org *Subject:* Re: [ceilometer] radosgw pollster Hi Florian,
have you tried different permissions for your ceilometer user in radosgw? According to the docs you need an admin user: https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.ht...
Our user has these caps: usage=read,write;metadata=read,write;users=read,write;buckets=read,write
We also had to add the requests-aws pip package to query radosgw from ceilometer: https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.htm...
Christian
Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch>>:
Hi Christian,
Am 2/26/19 um 11:00 AM schrieb Christian Zunker: > Hi Florian, > > which version of OpenStack are you using? > The radosgw metric names were different in some versions: > https://bugs.launchpad.net/ceilometer/+bug/1726458
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. As far as I am able to understand python it looks like the error is happening in polling.manager line 222:
https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manag...
But I do not understand why. I tried to enable debug logging but the error does not log any additional information. The poller is not even trying to reach/poll our RadosGWs. Looks like that manger is blocking those polls.
All the best, Florian
> > Christian > > Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann > <florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch> <mailto:florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch>>>: > > Hi, > > I failed to poll any usage data from our radosgw. I get > > 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling > pollster radosgw.containers.objects in the context of > radosgw_300s_pollsters > 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent > pollster radosgw.containers.objects from polling [<Project > description=, > domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True, > > [...] > > id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self': > u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source > radosgw_300s_pollsters anymore!: PollsterPermanentError > > Configurations like: > cat polling.yaml > --- > sources: > - name: radosgw_300s_pollsters > interval: 300 > meters: > - radosgw.usage > - radosgw.objects > - radosgw.objects.size > - radosgw.objects.containers > - radosgw.containers.objects > - radosgw.containers.objects.size > > > Also tried radosgw.api.requests instead of radowsgw.usage. > > ceilometer.conf > [...] > [service_types] > radosgw = object-store > > [rgw_admin_credentials] > access_key = xxxxx0Z0xxxxxxxxxxxx > secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx > > [rgw_client] > implicit_tenants = true > > Endpoints: > | xxxxxxx | region | swift | object-store | True | admin > | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | > | xxxxxxx | region | swift | object-store | True | > internal > | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s | > | xxxxxxx | region | swift | object-store | True | public > | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s | > > Ceilometer user: > { > "user_id": "ceilometer", > "display_name": "ceilometer", > "email": "", > "suspended": 0, > "max_buckets": 1000, > "auid": 0, > "subusers": [], > "keys": [ > { > "user": "ceilometer", > "access_key": "xxxxxxxxxxxxxxxxxx", > "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx" > } > ], > "swift_keys": [], > "caps": [ > { > "type": "buckets", > "perm": "read" > }, > { > "type": "metadata", > "perm": "read" > }, > { > "type": "usage", > "perm": "read" > }, > { > "type": "users", > "perm": "read" > } > ], > "op_mask": "read, write, delete", > "default_placement": "", > "placement_tags": [], > "bucket_quota": { > "enabled": false, > "check_on_raw": false, > "max_size": -1, > "max_size_kb": 0, > "max_objects": -1 > }, > "user_quota": { > "enabled": false, > "check_on_raw": false, > "max_size": -1, > "max_size_kb": 0, > "max_objects": -1 > }, > "temp_url_keys": [], > "type": "rgw" > } > > > radosgw config: > [client.rgw.xxxxxxxxxxx] > host = somehost > rgw frontends = "civetweb port=7480 num_threads=512" > rgw num rados handles = 8 > rgw thread pool size = 512 > rgw cache enabled = true > rgw dns name = s3.xxxxxx.xxx > rgw enable usage log = true > rgw usage log tick interval = 30 > rgw realm = public > rgw zonegroup = xxx > rgw zone = xxxxx > rgw resolve cname = False > rgw usage log flush threshold = 1024 > rgw usage max user shards = 1 > rgw usage max shards = 32 > rgw_keystone_url = https://keystone.xxxxxxxxxxxxx > rgw_keystone_admin_domain = default > rgw_keystone_admin_project = service > rgw_keystone_admin_user = swift > rgw_keystone_admin_password = > xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx > rgw_keystone_accepted_roles = member,_member_,admin > rgw_keystone_accepted_admin_roles = admin > rgw_keystone_api_version = 3 > rgw_keystone_verify_ssl = false > rgw_keystone_implicit_tenants = true > rgw_keystone_admin_tenant = default > rgw_keystone_revocation_interval = 0 > rgw_keystone_token_cache_size = 0 > rgw_s3_auth_use_keystone = true > rgw_max_attr_size = 1024 > rgw_max_attrs_num_in_req = 32 > rgw_max_attr_name_len = 64 > rgw_swift_account_in_url = true > rgw_swift_versioning_enabled = true > rgw_enable_apis = s3,swift,swift_auth,admin > rgw_swift_enforce_content_length = true > > > > > Any idea whats going on? > > All the best, > Florian > > >
-- EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich
tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.ch> web: http://www.everyware.ch
-- EveryWare AG Florian Engelmann Senior UNIX Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: mailto:florian.engelmann@everyware.ch web: http://www.everyware.ch
participants (3)
-
Christian Zunker
-
Engelmann Florian
-
Florian Engelmann