<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none"><!--P{margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr" style="font-size:10pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Hi Christian,</p>
<p><br>
</p>
<p>thank you for your feedback and help! Permissions are fine as I tried to poll the Endpoint successfully with curl and the user (key + secret) we created (and is configured in ceilometer.conf).</p>
<p>I saw the requests-aws is used in OSA and it is indeed missing in the kolla container (we use "source" not binary).</p>
<p><br>
</p>
<p><a href="https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-base/Dockerfile.j2">https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-base/Dockerfile.j2</a></p>
<p><br>
</p>
<p>I will build a new ceilometer container including requests-aws tomorrow to see if this fixes the problem.</p>
<p><br>
</p>
<p>All the best,</p>
<p>Florian<br>
</p>
<p><br>
</p>
<div style="color: rgb(33, 33, 33);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Christian Zunker <christian.zunker@codecentric.cloud><br>
<b>Sent:</b> Wednesday, February 27, 2019 9:09 AM<br>
<b>To:</b> Engelmann Florian<br>
<b>Cc:</b> openstack-discuss@lists.openstack.org<br>
<b>Subject:</b> Re: [ceilometer] radosgw pollster</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">Hi Florian,
<div><br>
</div>
<div>have you tried different permissions for your ceilometer user in radosgw?</div>
<div>According to the docs you need an admin user:</div>
<div><a href="https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#ceph-object-storage">https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#ceph-object-storage</a><br>
</div>
<div>Our user has these caps:</div>
usage=read,write;metadata=read,write;users=read,write;buckets=read,write</div>
</div>
<div><br>
</div>
<div>We also had to add the requests-aws pip package to query radosgw from ceilometer:</div>
<div><a href="https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.html">https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.html</a><br>
</div>
<div><br>
</div>
<div>Christian</div>
<div><br>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann <<a href="mailto:florian.engelmann@everyware.ch">florian.engelmann@everyware.ch</a>>:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
Hi Christian,<br>
<br>
Am 2/26/19 um 11:00 AM schrieb Christian Zunker:<br>
> Hi Florian,<br>
> <br>
> which version of OpenStack are you using?<br>
> The radosgw metric names were different in some versions: <br>
> <a href="https://bugs.launchpad.net/ceilometer/+bug/1726458" rel="noreferrer" target="_blank">
https://bugs.launchpad.net/ceilometer/+bug/1726458</a><br>
<br>
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error. <br>
As far as I am able to understand python it looks like the error is <br>
happening in polling.manager line 222:<br>
<br>
<a href="https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manager.py#L222" rel="noreferrer" target="_blank">https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manager.py#L222</a><br>
<br>
But I do not understand why. I tried to enable debug logging but the <br>
error does not log any additional information.<br>
The poller is not even trying to reach/poll our RadosGWs. Looks like <br>
that manger is blocking those polls.<br>
<br>
All the best,<br>
Florian<br>
<br>
<br>
> <br>
> Christian<br>
> <br>
> Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann <br>
> <<a href="mailto:florian.engelmann@everyware.ch" target="_blank">florian.engelmann@everyware.ch</a> <mailto:<a href="mailto:florian.engelmann@everyware.ch" target="_blank">florian.engelmann@everyware.ch</a>>>:<br>
> <br>
> Hi,<br>
> <br>
> I failed to poll any usage data from our radosgw. I get<br>
> <br>
> 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling<br>
> pollster radosgw.containers.objects in the context of<br>
> radosgw_300s_pollsters<br>
> 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent<br>
> pollster radosgw.containers.objects from polling [<Project<br>
> description=,<br>
> domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True,<br>
> <br>
> [...]<br>
> <br>
> id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self':<br>
> u'<a href="http://keystone-admin.service.xxxxxxx:35357/v3/projects" rel="noreferrer" target="_blank">http://keystone-admin.service.xxxxxxx:35357/v3/projects</a> on source<br>
> radosgw_300s_pollsters anymore!: PollsterPermanentError<br>
> <br>
> Configurations like:<br>
> cat polling.yaml<br>
> ---<br>
> sources:<br>
> - name: radosgw_300s_pollsters<br>
> interval: 300<br>
> meters:<br>
> - radosgw.usage<br>
> - radosgw.objects<br>
> - radosgw.objects.size<br>
> - radosgw.objects.containers<br>
> - radosgw.containers.objects<br>
> - radosgw.containers.objects.size<br>
> <br>
> <br>
> Also tried radosgw.api.requests instead of radowsgw.usage.<br>
> <br>
> ceilometer.conf<br>
> [...]<br>
> [service_types]<br>
> radosgw = object-store<br>
> <br>
> [rgw_admin_credentials]<br>
> access_key = xxxxx0Z0xxxxxxxxxxxx<br>
> secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx<br>
> <br>
> [rgw_client]<br>
> implicit_tenants = true<br>
> <br>
> Endpoints:<br>
> | xxxxxxx | region | swift | object-store | True | admin<br>
> | <a href="http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s" rel="noreferrer" target="_blank">
http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s</a> |<br>
> | xxxxxxx | region | swift | object-store | True |<br>
> internal<br>
> | <a href="http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s" rel="noreferrer" target="_blank">
http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s</a> |<br>
> | xxxxxxx | region | swift | object-store | True | public<br>
> | <a href="https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s" rel="noreferrer" target="_blank">
https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s</a> |<br>
> <br>
> Ceilometer user:<br>
> {<br>
> "user_id": "ceilometer",<br>
> "display_name": "ceilometer",<br>
> "email": "",<br>
> "suspended": 0,<br>
> "max_buckets": 1000,<br>
> "auid": 0,<br>
> "subusers": [],<br>
> "keys": [<br>
> {<br>
> "user": "ceilometer",<br>
> "access_key": "xxxxxxxxxxxxxxxxxx",<br>
> "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx"<br>
> }<br>
> ],<br>
> "swift_keys": [],<br>
> "caps": [<br>
> {<br>
> "type": "buckets",<br>
> "perm": "read"<br>
> },<br>
> {<br>
> "type": "metadata",<br>
> "perm": "read"<br>
> },<br>
> {<br>
> "type": "usage",<br>
> "perm": "read"<br>
> },<br>
> {<br>
> "type": "users",<br>
> "perm": "read"<br>
> }<br>
> ],<br>
> "op_mask": "read, write, delete",<br>
> "default_placement": "",<br>
> "placement_tags": [],<br>
> "bucket_quota": {<br>
> "enabled": false,<br>
> "check_on_raw": false,<br>
> "max_size": -1,<br>
> "max_size_kb": 0,<br>
> "max_objects": -1<br>
> },<br>
> "user_quota": {<br>
> "enabled": false,<br>
> "check_on_raw": false,<br>
> "max_size": -1,<br>
> "max_size_kb": 0,<br>
> "max_objects": -1<br>
> },<br>
> "temp_url_keys": [],<br>
> "type": "rgw"<br>
> }<br>
> <br>
> <br>
> radosgw config:<br>
> [client.rgw.xxxxxxxxxxx]<br>
> host = somehost<br>
> rgw frontends = "civetweb port=7480 num_threads=512"<br>
> rgw num rados handles = 8<br>
> rgw thread pool size = 512<br>
> rgw cache enabled = true<br>
> rgw dns name = s3.xxxxxx.xxx<br>
> rgw enable usage log = true<br>
> rgw usage log tick interval = 30<br>
> rgw realm = public<br>
> rgw zonegroup = xxx<br>
> rgw zone = xxxxx<br>
> rgw resolve cname = False<br>
> rgw usage log flush threshold = 1024<br>
> rgw usage max user shards = 1<br>
> rgw usage max shards = 32<br>
> rgw_keystone_url = <a href="https://keystone.xxxxxxxxxxxxx" rel="noreferrer" target="_blank">
https://keystone.xxxxxxxxxxxxx</a><br>
> rgw_keystone_admin_domain = default<br>
> rgw_keystone_admin_project = service<br>
> rgw_keystone_admin_user = swift<br>
> rgw_keystone_admin_password =<br>
> xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx<br>
> rgw_keystone_accepted_roles = member,_member_,admin<br>
> rgw_keystone_accepted_admin_roles = admin<br>
> rgw_keystone_api_version = 3<br>
> rgw_keystone_verify_ssl = false<br>
> rgw_keystone_implicit_tenants = true<br>
> rgw_keystone_admin_tenant = default<br>
> rgw_keystone_revocation_interval = 0<br>
> rgw_keystone_token_cache_size = 0<br>
> rgw_s3_auth_use_keystone = true<br>
> rgw_max_attr_size = 1024<br>
> rgw_max_attrs_num_in_req = 32<br>
> rgw_max_attr_name_len = 64<br>
> rgw_swift_account_in_url = true<br>
> rgw_swift_versioning_enabled = true<br>
> rgw_enable_apis = s3,swift,swift_auth,admin<br>
> rgw_swift_enforce_content_length = true<br>
> <br>
> <br>
> <br>
> <br>
> Any idea whats going on?<br>
> <br>
> All the best,<br>
> Florian<br>
> <br>
> <br>
> <br>
<br>
-- <br>
<br>
EveryWare AG<br>
Florian Engelmann<br>
Senior UNIX Systems Engineer<br>
Zurlindenstrasse 52a<br>
CH-8003 Zürich<br>
<br>
tel: +41 44 466 60 00<br>
fax: +41 44 466 60 10<br>
mail: mailto:<a href="mailto:florian.engelmann@everyware.ch" target="_blank">florian.engelmann@everyware.ch</a><br>
web: <a href="http://www.everyware.ch" rel="noreferrer" target="_blank">http://www.everyware.ch</a><br>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
</div>
</div>
</div>
</div>
</body>
</html>