RE: [Ceilometer] RADOS GW metrics : cannot get radosgw.objects.size metric

28 Oct 2022

      I tried "admin/usage" API instead of the "admin/bucket?stats=True" but the returned JSON does not contain information about bucket objects size.

So, I will keep on with “admin/bucket” and try to do the aggregations I need at gnocchi level.

Thanks again for your help !

JF

From: Rafael Weingärtner <rafaelweingartner@gmail.com>
Sent: vendredi, 28 octobre 2022 14:59
To: Taltavull Jean-François <jean-francois.taltavull@elca.ch>
Cc: openstack-discuss <openstack-discuss@lists.openstack.org>
Subject: Re: [Ceilometer] RADOS GW metrics : cannot get radosgw.objects.size metric

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

I am not understanding. Your expression to obtain the value is ". | value['usage'] | value.get('rgw.main',{'size':0}) | value['size']". That assumes a response with a "value" entry in the JSON; then, you get the 'rgw.main' attribute, and then, you get the size.

So, you are working with samples that are similar to the following:
```
{
   "bucket":"huge",
   "num_shards":11,
   "tenant":"08bb8ee9c5bd41248025268ee1aea481",
   "zonegroup":"d28c435f-57a5-49ca-91e8-481a2ced1f18",
   "placement_rule":"default-placement",
   "explicit_placement":{
      "data_pool":"",
      "data_extra_pool":"",
      "index_pool":""
   },
   "id":"ba604862-46ad-4cf1-a554-7da4e7168ac3.27108481.3",
   "marker":"ba604862-46ad-4cf1-a554-7da4e7168ac3.27108481.3",
   "index_type":"Normal",
   "owner":"08bb8ee9c5bd41248025268ee1aea481$08bb8ee9c5bd41248025268ee1aea481",
   "ver":"0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#2,8#1,9#1,10#1",
   "master_ver":"0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
   "mtime":"2022-10-26T11:32:05.185527Z",
   "creation_time":"2022-10-26T11:32:05.181022Z",
   "max_marker":"0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#",
   "usage":{
      "rgw.main":{
         "size":8461984,
         "size_actual":8462336,
         "size_utilized":8461984,
         "size_kb":8264,
         "size_kb_actual":8264,
         "size_kb_utilized":8264,
         "num_objects":1
      }
   },
   "bucket_quota":{
      "enabled":false,
      "check_on_raw":true,
      "max_size":-1,
      "max_size_kb":0,
      "max_objects":-1
   }
}
```

These samples are probably coming from an API with "usage" JSON attribute, and that is why it works with your expression.

To answer your initial question, Ceilometer dynamic pollster will work with whatever you have in the response. If data comes in a bucket fashion, each sample is going to represent a bucket. If you want to group/aggregate that data in a project/tenant fashion you might need to do some working. Either using a different API, or doing some groupby in Gnocchi with the aggregates API.

Furthermore, what about the "admin/usage" instead of the "admin/bucket?stats=True" . The admin API will bring data grouped in a user fashion.

On Fri, Oct 28, 2022 at 9:46 AM Taltavull Jean-François <jean-francois.taltavull@elca.ch<mailto:jean-francois.taltavull@elca.ch>> wrote:
See below. Hope this will help !

[[{'bucket': 'huge', 'num_shards': 11, 'tenant': '08bb8ee9c5bd41248025268ee1aea481', 'zonegroup': 'd28c435f-57a5-49ca-91e8-481a2ced1f18', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27108481.3', 'marker': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27108481.3', 'index_type': 'Normal', 'owner': '08bb8ee9c5bd41248025268ee1aea481$08bb8ee9c5bd41248025268ee1aea481', 'ver': '0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#2,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2022-10-26T11:32:05.185527Z', 'creation_time': '2022-10-26T11:32:05.181022Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 8461984, 'size_actual': 8462336, 'size_utilized': 8461984, 'size_kb': 8264, 'size_kb_actual': 8264, 'size_kb_utilized': 8264, 'num_objects': 1}}, 'bucket_quota': {'enabled': False, 'check_on_raw': True, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}}, {'bucket': 'empty', 'num_shards': 11, 'tenant': '08bb8ee9c5bd41248025268ee1aea481', 'zonegroup': 'd28c435f-57a5-49ca-91e8-481a2ced1f18', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27142035.4', 'marker': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27142035.4', 'index_type': 'Normal', 'owner': '08bb8ee9c5bd41248025268ee1aea481$08bb8ee9c5bd41248025268ee1aea481', 'ver': '0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2022-10-26T11:31:40.229337Z', 'creation_time': '2022-10-26T11:31:40.224401Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {}, 'bucket_quota': {'enabled': False, 'check_on_raw': True, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}}, {'bucket': 'photos', 'num_shards': 11, 'tenant': '08bb8ee9c5bd41248025268ee1aea481', 'zonegroup': 'd28c435f-57a5-49ca-91e8-481a2ced1f18', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27108481.2', 'marker': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27108481.2', 'index_type': 'Normal', 'owner': '08bb8ee9c5bd41248025268ee1aea481$08bb8ee9c5bd41248025268ee1aea481', 'ver': '0#2,1#1,2#1,3#3,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2022-10-24T11:54:18.320141Z', 'creation_time': '2022-10-24T11:54:18.315194Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 14, 'size_actual': 4096, 'size_utilized': 14, 'size_kb': 1, 'size_kb_actual': 4, 'size_kb_utilized': 1, 'num_objects': 1}}, 'bucket_quota': {'enabled': False, 'check_on_raw': True, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}}, {'bucket': 'big', 'num_shards': 11, 'tenant': '08bb8ee9c5bd41248025268ee1aea481', 'zonegroup': 'd28c435f-57a5-49ca-91e8-481a2ced1f18', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27100595.1', 'marker': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27100595.1', 'index_type': 'Normal', 'owner': '08bb8ee9c5bd41248025268ee1aea481$08bb8ee9c5bd41248025268ee1aea481', 'ver': '0#2,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2022-10-24T13:28:45.864925Z', 'creation_time': '2022-10-24T13:28:45.860346Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 249, 'size_actual': 4096, 'size_utilized': 249, 'size_kb': 1, 'size_kb_actual': 4, 'size_kb_utilized': 1, 'num_objects': 1}}, 'bucket_quota': {'enabled': False, 'check_on_raw': True, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}}]

From: Rafael Weingärtner <rafaelweingartner@gmail.com<mailto:rafaelweingartner@gmail.com>>
Sent: vendredi, 28 octobre 2022 12:27
To: Taltavull Jean-François <jean-francois.taltavull@elca.ch<mailto:jean-francois.taltavull@elca.ch>>
Cc: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>>
Subject: Re: [Ceilometer] RADOS GW metrics : cannot get radosgw.objects.size metric

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

Can you show us the json you are trying to process with Ceilometer? Then,we can move on from there. You can post here a minimalistic version of the json output.

On Fri, Oct 28, 2022 at 4:32 AM Taltavull Jean-François <jean-francois.taltavull@elca.ch<mailto:jean-francois.taltavull@elca.ch>> wrote:
Hello,

I can ask the question another way: what's the difference between 'radosgw.containers.objects.size' and 'radosgw.objects.size' metrics ?

Thanks,

JF
...
-----Original Message-----
From: Taltavull Jean-François
Sent: lundi, 24 octobre 2022 16:26
To: openstack-discuss <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>>
Subject: [Ceilometer] RADOS GW metrics : cannot get radosgw.objects.size
metric
Hello,
I'm trying to get the 'radosgw.objects.size' metric, that is the total bucket
objects size per tenant. I expected to get one sample per tenant but I get one
sample per bucket instead, as with the 'rados.containers.objects.size' metric.
Here is my pollster definition:
'''
- name: "radosgw.objects.size"
  sample_type: "gauge"
  unit: "B"
  value_attribute: ". | value['usage'] | value.get('rgw.main',{'size':0}) |
value['size']"
  url_path: "FQDN/admin/bucket?stats=True"
  module: "awsauth"
  authentication_object: "S3Auth"
  authentication_parameters: my_access_key,my_secret_key,FQDN
  user_id_attribute: "owner | value.split('$') | value[0]"
  project_id_attribute: "tenant"
  resource_id_attribute: "id"
'''
I tried with "resource_id_attribute: "tenant" but it does not work better.
Any idea ? Is there something wrong in the pollster definition ?
Regards,
Jean-Francois
--
Rafael Weingärtner

--
Rafael Weingärtner