Open Stack

I tried "admin/usage" API instead of the "admin/bucket?stats=True" but the returned JSON does not contain information about bucket objects size.

So, I will keep on with “admin/bucket” and try to do the aggregations I need at gnocchi level.

Thanks again for your help !

JF

From: Rafael Weingärtner <rafaelweingartner at gmail.com>
Sent: vendredi, 28 octobre 2022 14:59
To: Taltavull Jean-François <jean-francois.taltavull at elca.ch>
Cc: openstack-discuss <openstack-discuss at lists.openstack.org>
Subject: Re: [Ceilometer] RADOS GW metrics : cannot get radosgw.objects.size metric



EXTERNAL MESSAGE - This email comes from outside ELCA companies.

I am not understanding. Your expression to obtain the value is ". | value['usage'] | value.get('rgw.main',{'size':0}) | value['size']". That assumes a response with a "value" entry in the JSON; then, you get the 'rgw.main' attribute, and then, you get the size.



So, you are working with samples that are similar to the following:
```
{
   "bucket":"huge",
   "num_shards":11,
   "tenant":"08bb8ee9c5bd41248025268ee1aea481",
   "zonegroup":"d28c435f-57a5-49ca-91e8-481a2ced1f18",
   "placement_rule":"default-placement",
   "explicit_placement":{
      "data_pool":"",
      "data_extra_pool":"",
      "index_pool":""
   },
   "id":"ba604862-46ad-4cf1-a554-7da4e7168ac3.27108481.3",
   "marker":"ba604862-46ad-4cf1-a554-7da4e7168ac3.27108481.3",
   "index_type":"Normal",
   "owner":"08bb8ee9c5bd41248025268ee1aea481$08bb8ee9c5bd41248025268ee1aea481",
   "ver":"0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#2,8#1,9#1,10#1",
   "master_ver":"0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0",
   "mtime":"2022-10-26T11:32:05.185527Z",
   "creation_time":"2022-10-26T11:32:05.181022Z",
   "max_marker":"0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#",
   "usage":{
      "rgw.main":{
         "size":8461984,
         "size_actual":8462336,
         "size_utilized":8461984,
         "size_kb":8264,
         "size_kb_actual":8264,
         "size_kb_utilized":8264,
         "num_objects":1
      }
   },
   "bucket_quota":{
      "enabled":false,
      "check_on_raw":true,
      "max_size":-1,
      "max_size_kb":0,
      "max_objects":-1
   }
}
```

These samples are probably coming from an API with "usage" JSON attribute, and that is why it works with your expression.

To answer your initial question, Ceilometer dynamic pollster will work with whatever you have in the response. If data comes in a bucket fashion, each sample is going to represent a bucket. If you want to group/aggregate that data in a project/tenant fashion you might need to do some working. Either using a different API, or doing some groupby in Gnocchi with the aggregates API.

Furthermore, what about the "admin/usage" instead of the "admin/bucket?stats=True" . The admin API will bring data grouped in a user fashion.

On Fri, Oct 28, 2022 at 9:46 AM Taltavull Jean-François <jean-francois.taltavull at elca.ch<mailto:jean-francois.taltavull at elca.ch>> wrote:
See below. Hope this will help !

[[{'bucket': 'huge', 'num_shards': 11, 'tenant': '08bb8ee9c5bd41248025268ee1aea481', 'zonegroup': 'd28c435f-57a5-49ca-91e8-481a2ced1f18', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27108481.3', 'marker': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27108481.3', 'index_type': 'Normal', 'owner': '08bb8ee9c5bd41248025268ee1aea481$08bb8ee9c5bd41248025268ee1aea481', 'ver': '0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#2,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2022-10-26T11:32:05.185527Z', 'creation_time': '2022-10-26T11:32:05.181022Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 8461984, 'size_actual': 8462336, 'size_utilized': 8461984, 'size_kb': 8264, 'size_kb_actual': 8264, 'size_kb_utilized': 8264, 'num_objects': 1}}, 'bucket_quota': {'enabled': False, 'check_on_raw': True, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}}, {'bucket': 'empty', 'num_shards': 11, 'tenant': '08bb8ee9c5bd41248025268ee1aea481', 'zonegroup': 'd28c435f-57a5-49ca-91e8-481a2ced1f18', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27142035.4', 'marker': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27142035.4', 'index_type': 'Normal', 'owner': '08bb8ee9c5bd41248025268ee1aea481$08bb8ee9c5bd41248025268ee1aea481', 'ver': '0#1,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2022-10-26T11:31:40.229337Z', 'creation_time': '2022-10-26T11:31:40.224401Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {}, 'bucket_quota': {'enabled': False, 'check_on_raw': True, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}}, {'bucket': 'photos', 'num_shards': 11, 'tenant': '08bb8ee9c5bd41248025268ee1aea481', 'zonegroup': 'd28c435f-57a5-49ca-91e8-481a2ced1f18', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27108481.2', 'marker': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27108481.2', 'index_type': 'Normal', 'owner': '08bb8ee9c5bd41248025268ee1aea481$08bb8ee9c5bd41248025268ee1aea481', 'ver': '0#2,1#1,2#1,3#3,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2022-10-24T11:54:18.320141Z', 'creation_time': '2022-10-24T11:54:18.315194Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 14, 'size_actual': 4096, 'size_utilized': 14, 'size_kb': 1, 'size_kb_actual': 4, 'size_kb_utilized': 1, 'num_objects': 1}}, 'bucket_quota': {'enabled': False, 'check_on_raw': True, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}}, {'bucket': 'big', 'num_shards': 11, 'tenant': '08bb8ee9c5bd41248025268ee1aea481', 'zonegroup': 'd28c435f-57a5-49ca-91e8-481a2ced1f18', 'placement_rule': 'default-placement', 'explicit_placement': {'data_pool': '', 'data_extra_pool': '', 'index_pool': ''}, 'id': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27100595.1', 'marker': 'ba604862-46ad-4cf1-a554-7da4e7168ac3.27100595.1', 'index_type': 'Normal', 'owner': '08bb8ee9c5bd41248025268ee1aea481$08bb8ee9c5bd41248025268ee1aea481', 'ver': '0#2,1#1,2#1,3#1,4#1,5#1,6#1,7#1,8#1,9#1,10#1', 'master_ver': '0#0,1#0,2#0,3#0,4#0,5#0,6#0,7#0,8#0,9#0,10#0', 'mtime': '2022-10-24T13:28:45.864925Z', 'creation_time': '2022-10-24T13:28:45.860346Z', 'max_marker': '0#,1#,2#,3#,4#,5#,6#,7#,8#,9#,10#', 'usage': {'rgw.main': {'size': 249, 'size_actual': 4096, 'size_utilized': 249, 'size_kb': 1, 'size_kb_actual': 4, 'size_kb_utilized': 1, 'num_objects': 1}}, 'bucket_quota': {'enabled': False, 'check_on_raw': True, 'max_size': -1, 'max_size_kb': 0, 'max_objects': -1}}]


From: Rafael Weingärtner <rafaelweingartner at gmail.com<mailto:rafaelweingartner at gmail.com>>
Sent: vendredi, 28 octobre 2022 12:27
To: Taltavull Jean-François <jean-francois.taltavull at elca.ch<mailto:jean-francois.taltavull at elca.ch>>
Cc: openstack-discuss <openstack-discuss at lists.openstack.org<mailto:openstack-discuss at lists.openstack.org>>
Subject: Re: [Ceilometer] RADOS GW metrics : cannot get radosgw.objects.size metric



EXTERNAL MESSAGE - This email comes from outside ELCA companies.

Can you show us the json you are trying to process with Ceilometer? Then,we can move on from there. You can post here a minimalistic version of the json output.

On Fri, Oct 28, 2022 at 4:32 AM Taltavull Jean-François <jean-francois.taltavull at elca.ch<mailto:jean-francois.taltavull at elca.ch>> wrote:
Hello,

I can ask the question another way: what's the difference between 'radosgw.containers.objects.size' and 'radosgw.objects.size' metrics ?

Thanks,

JF

> -----Original Message-----
> From: Taltavull Jean-François
> Sent: lundi, 24 octobre 2022 16:26
> To: openstack-discuss <openstack-discuss at lists.openstack.org<mailto:openstack-discuss at lists.openstack.org>>
> Subject: [Ceilometer] RADOS GW metrics : cannot get radosgw.objects.size
> metric
>
> Hello,
>
> I'm trying to get the 'radosgw.objects.size' metric, that is the total bucket
> objects size per tenant. I expected to get one sample per tenant but I get one
> sample per bucket instead, as with the 'rados.containers.objects.size' metric.
>
> Here is my pollster definition:
> '''
> - name: "radosgw.objects.size"
>   sample_type: "gauge"
>   unit: "B"
>   value_attribute: ". | value['usage'] | value.get('rgw.main',{'size':0}) |
> value['size']"
>   url_path: "FQDN/admin/bucket?stats=True"
>   module: "awsauth"
>   authentication_object: "S3Auth"
>   authentication_parameters: my_access_key,my_secret_key,FQDN
>   user_id_attribute: "owner | value.split('$') | value[0]"
>   project_id_attribute: "tenant"
>   resource_id_attribute: "id"
> '''
>
> I tried with "resource_id_attribute: "tenant" but it does not work better.
>
> Any idea ? Is there something wrong in the pollster definition ?
>
> Regards,
> Jean-Francois


--
Rafael Weingärtner


--
Rafael Weingärtner
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20221028/3976efb3/attachment-0001.htm>