Re: [ceilometer] radosgw pollster

27 Feb 2019

      Hi Christian,
looks like a hit:
https://github.com/openstack/ceilometer/commit/c9eb2d44df7cafde1294123d66445...
You made my day!
I will test tomorrow and report back!

All the best,
Florian
________________________________
From: Engelmann Florian florian.engelmann@everyware.ch
Sent: Wednesday, February 27, 2019 8:33 PM
To: Christian Zunker
Cc: openstack-discuss@lists.openstack.org
Subject: Re: [ceilometer] radosgw pollster
Hi Christian,
thank you for your feedback and help! Permissions are fine as I tried to poll the Endpoint successfully with curl and the user (key + secret) we created (and is configured in ceilometer.conf).
I saw the requests-aws is used in OSA and it is indeed missing in the kolla container (we use "source" not binary).
https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-...
I will build a new ceilometer container including requests-aws tomorrow to see if this fixes the problem.
All the best,
Florian
________________________________
From: Christian Zunker christian.zunker@codecentric.cloud
Sent: Wednesday, February 27, 2019 9:09 AM
To: Engelmann Florian
Cc: openstack-discuss@lists.openstack.org
Subject: Re: [ceilometer] radosgw pollster
Hi Florian,
have you tried different permissions for your ceilometer user in radosgw?
According to the docs you need an admin user:
https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.ht...
Our user has these caps:
usage=read,write;metadata=read,write;users=read,write;buckets=read,write
We also had to add the requests-aws pip package to query radosgw from ceilometer:
https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.htm...
Christian
Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann <florian.engelmann@everyware.chmailto:florian.engelmann@everyware.ch>:
Hi Christian,
Am 2/26/19 um 11:00 AM schrieb Christian Zunker:
...
Hi Florian,
which version of OpenStack are you using?
The radosgw metric names were different in some versions:
https://bugs.launchpad.net/ceilometer/+bug/1726458
we do use Rocky and Ceilometer 11.0.1. I am still lost with that error.
As far as I am able to understand python it looks like the error is
happening in polling.manager line 222:
https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manag...
But I do not understand why. I tried to enable debug logging but the
error does not log any additional information.
The poller is not even trying to reach/poll our RadosGWs. Looks like
that manger is blocking those polls.
All the best,
Florian
...
Christian
Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann
<florian.engelmann@everyware.chmailto:florian.engelmann@everyware.ch <mailto:florian.engelmann@everyware.chmailto:florian.engelmann@everyware.ch>>:
Hi,

I failed to poll any usage data from our radosgw. I get

2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager [-] Polling
pollster radosgw.containers.objects in the context of
radosgw_300s_pollsters
2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager [-] Prevent
pollster radosgw.containers.objects from polling [<Project
description=,
domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True,

[...]

id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False, links={u'self':
u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on source
radosgw_300s_pollsters anymore!: PollsterPermanentError

Configurations like:
cat polling.yaml
---
sources:
      - name: radosgw_300s_pollsters
        interval: 300
        meters:
          - radosgw.usage
          - radosgw.objects
          - radosgw.objects.size
          - radosgw.objects.containers
          - radosgw.containers.objects
          - radosgw.containers.objects.size

Also tried radosgw.api.requests instead of radowsgw.usage.

ceilometer.conf
[...]
[service_types]
radosgw = object-store

[rgw_admin_credentials]
access_key = xxxxx0Z0xxxxxxxxxxxx
secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx

[rgw_client]
implicit_tenants = true

Endpoints:
| xxxxxxx | region | swift        | object-store    | True    | admin
   | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s  |
| xxxxxxx | region | swift        | object-store    | True    |
internal
   | http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s  |
| xxxxxxx | region | swift        | object-store    | True    | public
   | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s       |

Ceilometer user:
{
      "user_id": "ceilometer",
      "display_name": "ceilometer",
      "email": "",
      "suspended": 0,
      "max_buckets": 1000,
      "auid": 0,
      "subusers": [],
      "keys": [
          {
              "user": "ceilometer",
              "access_key": "xxxxxxxxxxxxxxxxxx",
              "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx"
          }
      ],
      "swift_keys": [],
      "caps": [
          {
              "type": "buckets",
              "perm": "read"
          },
          {
              "type": "metadata",
              "perm": "read"
          },
          {
              "type": "usage",
              "perm": "read"
          },
          {
              "type": "users",
              "perm": "read"
          }
      ],
      "op_mask": "read, write, delete",
      "default_placement": "",
      "placement_tags": [],
      "bucket_quota": {
          "enabled": false,
          "check_on_raw": false,
          "max_size": -1,
          "max_size_kb": 0,
          "max_objects": -1
      },
      "user_quota": {
          "enabled": false,
          "check_on_raw": false,
          "max_size": -1,
          "max_size_kb": 0,
          "max_objects": -1
      },
      "temp_url_keys": [],
      "type": "rgw"
}

radosgw config:
[client.rgw.xxxxxxxxxxx]
host = somehost
rgw frontends = "civetweb port=7480 num_threads=512"
rgw num rados handles = 8
rgw thread pool size = 512
rgw cache enabled = true
rgw dns name = s3.xxxxxx.xxx
rgw enable usage log = true
rgw usage log tick interval = 30
rgw realm = public
rgw zonegroup = xxx
rgw zone = xxxxx
rgw resolve cname = False
rgw usage log flush threshold = 1024
rgw usage max user shards = 1
rgw usage max shards = 32
rgw_keystone_url = https://keystone.xxxxxxxxxxxxx
rgw_keystone_admin_domain = default
rgw_keystone_admin_project = service
rgw_keystone_admin_user = swift
rgw_keystone_admin_password =
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
rgw_keystone_accepted_roles = member,_member_,admin
rgw_keystone_accepted_admin_roles = admin
rgw_keystone_api_version = 3
rgw_keystone_verify_ssl = false
rgw_keystone_implicit_tenants = true
rgw_keystone_admin_tenant = default
rgw_keystone_revocation_interval = 0
rgw_keystone_token_cache_size = 0
rgw_s3_auth_use_keystone = true
rgw_max_attr_size = 1024
rgw_max_attrs_num_in_req = 32
rgw_max_attr_name_len = 64
rgw_swift_account_in_url = true
rgw_swift_versioning_enabled = true
rgw_enable_apis = s3,swift,swift_auth,admin
rgw_swift_enforce_content_length = true

Any idea whats going on?

All the best,
Florian

--
EveryWare AG
Florian Engelmann
Senior UNIX Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelmann@everyware.chmailto:florian.engelmann@everyware.ch
web: http://www.everyware.ch