<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Florian,<div><br></div><div>have you tried different permissions for your ceilometer user in radosgw?</div><div>According to the docs you need an admin user:</div><div><a href="https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#ceph-object-storage">https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#ceph-object-storage</a><br></div><div>Our user has these caps:</div>usage=read,write;metadata=read,write;users=read,write;buckets=read,write</div></div><div><br></div><div>We also had to add the requests-aws pip package to query radosgw from ceilometer:</div><div><a href="https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.html">https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.html</a><br></div><div><br></div><div>Christian</div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann <<a href="mailto:florian.engelmann@everyware.ch">florian.engelmann@everyware.ch</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Christian,<br>
<br>
Am 2/26/19 um 11:00 AM schrieb Christian Zunker:<br>
> Hi Florian,<br>
> <br>
> which version of OpenStack are you using?<br>
> The radosgw metric names were different in some versions: <br>
> <a href="https://bugs.launchpad.net/ceilometer/+bug/1726458" rel="noreferrer" target="_blank">https://bugs.launchpad.net/ceilometer/+bug/1726458</a><br>
<br>
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. <br>
As far as I am able to understand python it looks like the error is <br>
happening in polling.manager line 222:<br>
<br>
<a href="https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manager.py#L222" rel="noreferrer" target="_blank">https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manager.py#L222</a><br>
<br>
But I do not understand why. I tried to enable debug logging but the <br>
error does not log any additional information.<br>
The poller is not even trying to reach/poll our RadosGWs. Looks like <br>
that manger is blocking those polls.<br>
<br>
All the best,<br>
Florian<br>
<br>
<br>
> <br>
> Christian<br>
> <br>
> Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann <br>
> <<a href="mailto:florian.engelmann@everyware.ch" target="_blank">florian.engelmann@everyware.ch</a> <mailto:<a href="mailto:florian.engelmann@everyware.ch" target="_blank">florian.engelmann@everyware.ch</a>>>:<br>
> <br>
> Hi,<br>
> <br>
> I failed to poll any usage data from our radosgw. I get<br>
> <br>
> 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling<br>
> pollster radosgw.containers.objects in the context of<br>
> radosgw_300s_pollsters<br>
> 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent<br>
> pollster radosgw.containers.objects from polling [<Project<br>
> description=,<br>
> domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True,<br>
> <br>
> [...]<br>
> <br>
> id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self':<br>
> u'<a href="http://keystone-admin.service.xxxxxxx:35357/v3/projects" rel="noreferrer" target="_blank">http://keystone-admin.service.xxxxxxx:35357/v3/projects</a> on source<br>
> radosgw_300s_pollsters anymore!: PollsterPermanentError<br>
> <br>
> Configurations like:<br>
> cat polling.yaml<br>
> ---<br>
> sources:<br>
> - name: radosgw_300s_pollsters<br>
> interval: 300<br>
> meters:<br>
> - radosgw.usage<br>
> - radosgw.objects<br>
> - radosgw.objects.size<br>
> - radosgw.objects.containers<br>
> - radosgw.containers.objects<br>
> - radosgw.containers.objects.size<br>
> <br>
> <br>
> Also tried radosgw.api.requests instead of radowsgw.usage.<br>
> <br>
> ceilometer.conf<br>
> [...]<br>
> [service_types]<br>
> radosgw = object-store<br>
> <br>
> [rgw_admin_credentials]<br>
> access_key = xxxxx0Z0xxxxxxxxxxxx<br>
> secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx<br>
> <br>
> [rgw_client]<br>
> implicit_tenants = true<br>
> <br>
> Endpoints:<br>
> | xxxxxxx | region | swift | object-store | True | admin<br>
> | <a href="http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s" rel="noreferrer" target="_blank">http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s</a> |<br>
> | xxxxxxx | region | swift | object-store | True |<br>
> internal<br>
> | <a href="http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s" rel="noreferrer" target="_blank">http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s</a> |<br>
> | xxxxxxx | region | swift | object-store | True | public<br>
> | <a href="https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s" rel="noreferrer" target="_blank">https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s</a> |<br>
> <br>
> Ceilometer user:<br>
> {<br>
> "user_id": "ceilometer",<br>
> "display_name": "ceilometer",<br>
> "email": "",<br>
> "suspended": 0,<br>
> "max_buckets": 1000,<br>
> "auid": 0,<br>
> "subusers": [],<br>
> "keys": [<br>
> {<br>
> "user": "ceilometer",<br>
> "access_key": "xxxxxxxxxxxxxxxxxx",<br>
> "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx"<br>
> }<br>
> ],<br>
> "swift_keys": [],<br>
> "caps": [<br>
> {<br>
> "type": "buckets",<br>
> "perm": "read"<br>
> },<br>
> {<br>
> "type": "metadata",<br>
> "perm": "read"<br>
> },<br>
> {<br>
> "type": "usage",<br>
> "perm": "read"<br>
> },<br>
> {<br>
> "type": "users",<br>
> "perm": "read"<br>
> }<br>
> ],<br>
> "op_mask": "read, write, delete",<br>
> "default_placement": "",<br>
> "placement_tags": [],<br>
> "bucket_quota": {<br>
> "enabled": false,<br>
> "check_on_raw": false,<br>
> "max_size": -1,<br>
> "max_size_kb": 0,<br>
> "max_objects": -1<br>
> },<br>
> "user_quota": {<br>
> "enabled": false,<br>
> "check_on_raw": false,<br>
> "max_size": -1,<br>
> "max_size_kb": 0,<br>
> "max_objects": -1<br>
> },<br>
> "temp_url_keys": [],<br>
> "type": "rgw"<br>
> }<br>
> <br>
> <br>
> radosgw config:<br>
> [client.rgw.xxxxxxxxxxx]<br>
> host = somehost<br>
> rgw frontends = "civetweb port=7480 num_threads=512"<br>
> rgw num rados handles = 8<br>
> rgw thread pool size = 512<br>
> rgw cache enabled = true<br>
> rgw dns name = s3.xxxxxx.xxx<br>
> rgw enable usage log = true<br>
> rgw usage log tick interval = 30<br>
> rgw realm = public<br>
> rgw zonegroup = xxx<br>
> rgw zone = xxxxx<br>
> rgw resolve cname = False<br>
> rgw usage log flush threshold = 1024<br>
> rgw usage max user shards = 1<br>
> rgw usage max shards = 32<br>
> rgw_keystone_url = <a href="https://keystone.xxxxxxxxxxxxx" rel="noreferrer" target="_blank">https://keystone.xxxxxxxxxxxxx</a><br>
> rgw_keystone_admin_domain = default<br>
> rgw_keystone_admin_project = service<br>
> rgw_keystone_admin_user = swift<br>
> rgw_keystone_admin_password =<br>
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx<br>
> rgw_keystone_accepted_roles = member,_member_,admin<br>
> rgw_keystone_accepted_admin_roles = admin<br>
> rgw_keystone_api_version = 3<br>
> rgw_keystone_verify_ssl = false<br>
> rgw_keystone_implicit_tenants = true<br>
> rgw_keystone_admin_tenant = default<br>
> rgw_keystone_revocation_interval = 0<br>
> rgw_keystone_token_cache_size = 0<br>
> rgw_s3_auth_use_keystone = true<br>
> rgw_max_attr_size = 1024<br>
> rgw_max_attrs_num_in_req = 32<br>
> rgw_max_attr_name_len = 64<br>
> rgw_swift_account_in_url = true<br>
> rgw_swift_versioning_enabled = true<br>
> rgw_enable_apis = s3,swift,swift_auth,admin<br>
> rgw_swift_enforce_content_length = true<br>
> <br>
> <br>
> <br>
> <br>
> Any idea whats going on?<br>
> <br>
> All the best,<br>
> Florian<br>
> <br>
> <br>
> <br>
<br>
-- <br>
<br>
EveryWare AG<br>
Florian Engelmann<br>
Senior UNIX Systems Engineer<br>
Zurlindenstrasse 52a<br>
CH-8003 Zürich<br>
<br>
tel: +41 44 466 60 00<br>
fax: +41 44 466 60 10<br>
mail: mailto:<a href="mailto:florian.engelmann@everyware.ch" target="_blank">florian.engelmann@everyware.ch</a><br>
web: <a href="http://www.everyware.ch" rel="noreferrer" target="_blank">http://www.everyware.ch</a><br>
</blockquote></div><br clear="all"><div><br></div></div></div>