Hello,

I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer logs, that it’s actually loaded. But it looks like it was not triggered, I see no trace of ceilometer connection in Rados GW logs.

My definition:

- name: "dynamic.radosgw.usage" sample_type: "gauge" unit: "B" value_attribute: "total.size" url_path: http://<FQDN>/object-store/swift/v1/admin/usagehttp://%3cFQDN%3e/object-store/swift/v1/admin/usage module: "awsauth" authentication_object: "S3Auth" authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN> user_id_attribute: "admin" project_id_attribute: "admin" resource_id_attribute: "admin" response_entries_key: "summary"

Do I have to set an option in ceilometer.conf, or elsewhere, to get my Rados GW dynamic pollster triggered ?

-JF

From: Taltavull Jean-François Sent: lundi, 29 août 2022 18:41 To: 'Rafael Weingärtner' rafaelweingartner@gmail.com Cc: openstack-discuss openstack-discuss@lists.openstack.org Subject: RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

Thanks a lot for your quick answer, Rafael ! I will explore this approach.

Jean-Francois

From: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Sent: lundi, 29 août 2022 17:54 To: Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

You could use a different approach. You can use Dynamic pollster [1], and create your own mechanism to collect data, without needing to change Ceilometer code. Basically all hard-coded pollsters can be converted to a dynamic pollster that is defined in YML.

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollste...

On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> wrote: Hi All,

In our OpenStack deployment, API endpoints are defined by using URLs instead of port numbers and HAProxy forwards requests to the right bakend after having ACLed the URL.

In the case of our object-store service, based on RadosGW, the internal API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>https://%3cFQDN%3e/object-store/swift/v1/AUTH_%3ctenant_id%3e"

When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API with the object-store internal endpoint, the URL becomes https://<FQDN>/adminhttps://%3cFQDN%3e/admin, as shown by HAProxy logs. This URL does not match any API endpoint from HAProxy point of view. The line of code that rewrites the URL is this one: https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilomete...

What would you think of adding a mechanism based on new Ceilometer configuration option(s) to control the URL rewriting ?

Our deployment characteristics: - OpenStack release: Wallaby - Ceph and RadosGW version: 15.2.16 - deployment tool: OSA 23.2.1 and ceph-ansible

Best regards, Jean-Francois

-- Rafael Weingärtner

Thanks to your help, I am close to the goal. Dynamic pollster is loaded and triggered.

But I get a “Status[403] and reason [Forbidden]” in ceilometer logs while requesting admin/usage.

I’m not sure to understand well the auth mechanism. Are we talking about keystone credentials, ec2 credentials, Rados GW user ?...

For now, in testing phase, I use “authentication_parameters”, not barbican.

-JF

From: Rafael Weingärtner rafaelweingartner@gmail.com Sent: mardi, 30 août 2022 14:17 To: Taltavull Jean-François jean-francois.taltavull@elca.ch Cc: openstack-discuss openstack-discuss@lists.openstack.org Subject: Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

Yes, you will need to enable the metric/pollster to be processed. That is done via "polling.yml" file. Also, do not forget that you will need to configure Ceilometer to push this new metric. If you use Gnocchi as the backend, you will need to change/update the gnocchi resource YML file. That file maps resources and metrics in the Gnocchi backend. The configuration resides in Ceilometer. You can create/define new resource types and map them to specific metrics. It depends on how you structure your solution.

P.S. You do not need to use "authentication_parameters". You can use the barbican integration to avoid setting your credentials in a file.

On Tue, Aug 30, 2022 at 9:11 AM Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> wrote: Hello,

I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer logs, that it’s actually loaded. But it looks like it was not triggered, I see no trace of ceilometer connection in Rados GW logs.

My definition:

- name: "dynamic.radosgw.usage" sample_type: "gauge" unit: "B" value_attribute: "total.size" url_path: http://<FQDN>/object-store/swift/v1/admin/usagehttp://%3cFQDN%3e/object-store/swift/v1/admin/usage module: "awsauth" authentication_object: "S3Auth" authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN> user_id_attribute: "admin" project_id_attribute: "admin" resource_id_attribute: "admin" response_entries_key: "summary"

Do I have to set an option in ceilometer.conf, or elsewhere, to get my Rados GW dynamic pollster triggered ?

-JF

From: Taltavull Jean-François Sent: lundi, 29 août 2022 18:41 To: 'Rafael Weingärtner' <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

Thanks a lot for your quick answer, Rafael ! I will explore this approach.

Jean-Francois

From: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Sent: lundi, 29 août 2022 17:54 To: Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

You could use a different approach. You can use Dynamic pollster [1], and create your own mechanism to collect data, without needing to change Ceilometer code. Basically all hard-coded pollsters can be converted to a dynamic pollster that is defined in YML.

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollste...

On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> wrote: Hi All,

In our OpenStack deployment, API endpoints are defined by using URLs instead of port numbers and HAProxy forwards requests to the right bakend after having ACLed the URL.

In the case of our object-store service, based on RadosGW, the internal API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>https://%3cFQDN%3e/object-store/swift/v1/AUTH_%3ctenant_id%3e"

When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API with the object-store internal endpoint, the URL becomes https://<FQDN>/adminhttps://%3cFQDN%3e/admin, as shown by HAProxy logs. This URL does not match any API endpoint from HAProxy point of view. The line of code that rewrites the URL is this one: https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilomete...

What would you think of adding a mechanism based on new Ceilometer configuration option(s) to control the URL rewriting ?

Our deployment characteristics: - OpenStack release: Wallaby - Ceph and RadosGW version: 15.2.16 - deployment tool: OSA 23.2.1 and ceph-ansible

Best regards, Jean-Francois

-- Rafael Weingärtner

-- Rafael Weingärtner

Jean, there are two problems with the Ceilometer. I just opened the patches to resolve it: - https://review.opendev.org/c/openstack/ceilometer/+/856305 - https://review.opendev.org/c/openstack/ceilometer/+/856304

Without these patches, you might have problems to use Ceilometer with Non-OpenStack dynamic pollsters and barbican credentials.

On Wed, Aug 31, 2022 at 3:55 PM Rafael Weingärtner < rafaelweingartner@gmail.com> wrote:

It is the RGW user that you have. This user must have the role that is needed to access the usage feature in RGW. If I am not mistaken, it required an admin user.

On Wed, Aug 31, 2022 at 1:54 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

...

Thanks to your help, I am close to the goal. Dynamic pollster is loaded and triggered.

But I get a “Status[403] and reason [Forbidden]” in ceilometer logs while requesting admin/usage.

I’m not sure to understand well the auth mechanism. Are we talking about keystone credentials, ec2 credentials, Rados GW user ?...

For now, in testing phase, I use “authentication_parameters”, not barbican.

-JF

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mardi, 30 août 2022 14:17 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Yes, you will need to enable the metric/pollster to be processed. That is done via "polling.yml" file. Also, do not forget that you will need to configure Ceilometer to push this new metric. If you use Gnocchi as the backend, you will need to change/update the gnocchi resource YML file. That file maps resources and metrics in the Gnocchi backend. The configuration resides in Ceilometer. You can create/define new resource types and map them to specific metrics. It depends on how you structure your solution.

P.S. You do not need to use "authentication_parameters". You can use the barbican integration to avoid setting your credentials in a file.

On Tue, Aug 30, 2022 at 9:11 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello,

I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer logs, that it’s actually loaded. But it looks like it was not triggered, I see no trace of ceilometer connection in Rados GW logs.

My definition:

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/swift/v1/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN>

  user_id_attribute: "admin"

  project_id_attribute: "admin"

  resource_id_attribute: "admin"

  response_entries_key: "summary"

Do I have to set an option in ceilometer.conf, or elsewhere, to get my Rados GW dynamic pollster triggered ?

-JF

*From:* Taltavull Jean-François *Sent:* lundi, 29 août 2022 18:41 *To:* 'Rafael Weingärtner' rafaelweingartner@gmail.com *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

Thanks a lot for your quick answer, Rafael !

I will explore this approach.

Jean-Francois

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* lundi, 29 août 2022 17:54 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

You could use a different approach. You can use Dynamic pollster [1], and create your own mechanism to collect data, without needing to change Ceilometer code. Basically all hard-coded pollsters can be converted to a dynamic pollster that is defined in YML.

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollste...

On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hi All,

In our OpenStack deployment, API endpoints are defined by using URLs instead of port numbers and HAProxy forwards requests to the right bakend after having ACLed the URL.

In the case of our object-store service, based on RadosGW, the internal API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>"

When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API with the object-store internal endpoint, the URL becomes https://<FQDN>/admin, as shown by HAProxy logs. This URL does not match any API endpoint from HAProxy point of view. The line of code that rewrites the URL is this one: https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilomete...

What would you think of adding a mechanism based on new Ceilometer configuration option(s) to control the URL rewriting ?

Our deployment characteristics:

• OpenStack release: Wallaby
• Ceph and RadosGW version: 15.2.16
• deployment tool: OSA 23.2.1 and ceph-ansible

Best regards, Jean-Francois

--

Rafael Weingärtner

--

Rafael Weingärtner

-- Rafael Weingärtner

Can you show your YML configuration? Also, did you install the AWS authentication module in the container/host where Ceilometer central is running?

On Mon, Sep 26, 2022 at 12:58 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello Rafael,

Thanks for the information about ceilometer patches but for now I’m testing with the credentials in the dynamic pollster config file. I will use barbican when I push all this to production.

The keystone authentication performed by the rados gw with the credentials provided by ceilometer still does not work. I wonder if this could be a S3 signature version issue on ceilometer side, that is on S3 client side. This kind of issue exists with the s3 client “s3cmd” and you have to add “—signature-v2” so that “s3cmd” works well.

What do you think ? Do you know which version of S3 signature ceilometer uses while authenticating ?

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 7 septembre 2022 19:23 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Jean, there are two problems with the Ceilometer. I just opened the patches to resolve it:

• https://review.opendev.org/c/openstack/ceilometer/+/856305

• https://review.opendev.org/c/openstack/ceilometer/+/856304

Without these patches, you might have problems to use Ceilometer with Non-OpenStack dynamic pollsters and barbican credentials.

On Wed, Aug 31, 2022 at 3:55 PM Rafael Weingärtner < rafaelweingartner@gmail.com> wrote:

It is the RGW user that you have. This user must have the role that is needed to access the usage feature in RGW. If I am not mistaken, it required an admin user.

On Wed, Aug 31, 2022 at 1:54 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Thanks to your help, I am close to the goal. Dynamic pollster is loaded and triggered.

But I get a “Status[403] and reason [Forbidden]” in ceilometer logs while requesting admin/usage.

I’m not sure to understand well the auth mechanism. Are we talking about keystone credentials, ec2 credentials, Rados GW user ?...

For now, in testing phase, I use “authentication_parameters”, not barbican.

-JF

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mardi, 30 août 2022 14:17 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Yes, you will need to enable the metric/pollster to be processed. That is done via "polling.yml" file. Also, do not forget that you will need to configure Ceilometer to push this new metric. If you use Gnocchi as the backend, you will need to change/update the gnocchi resource YML file. That file maps resources and metrics in the Gnocchi backend. The configuration resides in Ceilometer. You can create/define new resource types and map them to specific metrics. It depends on how you structure your solution.

P.S. You do not need to use "authentication_parameters". You can use the barbican integration to avoid setting your credentials in a file.

On Tue, Aug 30, 2022 at 9:11 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello,

I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer logs, that it’s actually loaded. But it looks like it was not triggered, I see no trace of ceilometer connection in Rados GW logs.

My definition:

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/swift/v1/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN>

  user_id_attribute: "admin"

  project_id_attribute: "admin"

  resource_id_attribute: "admin"

  response_entries_key: "summary"

Do I have to set an option in ceilometer.conf, or elsewhere, to get my Rados GW dynamic pollster triggered ?

-JF

*From:* Taltavull Jean-François *Sent:* lundi, 29 août 2022 18:41 *To:* 'Rafael Weingärtner' rafaelweingartner@gmail.com *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

Thanks a lot for your quick answer, Rafael !

I will explore this approach.

Jean-Francois

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* lundi, 29 août 2022 17:54 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

You could use a different approach. You can use Dynamic pollster [1], and create your own mechanism to collect data, without needing to change Ceilometer code. Basically all hard-coded pollsters can be converted to a dynamic pollster that is defined in YML.

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollste...

On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hi All,

In our OpenStack deployment, API endpoints are defined by using URLs instead of port numbers and HAProxy forwards requests to the right bakend after having ACLed the URL.

In the case of our object-store service, based on RadosGW, the internal API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>"

When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API with the object-store internal endpoint, the URL becomes https://<FQDN>/admin, as shown by HAProxy logs. This URL does not match any API endpoint from HAProxy point of view. The line of code that rewrites the URL is this one: https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilomete...

What would you think of adding a mechanism based on new Ceilometer configuration option(s) to control the URL rewriting ?

Our deployment characteristics:

• OpenStack release: Wallaby
• Ceph and RadosGW version: 15.2.16
• deployment tool: OSA 23.2.1 and ceph-ansible

Best regards, Jean-Francois

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

I think that the last parameter "<FQDN>/object-store/", should be only " <FQDN>". Can you test it?

You are using EC2 credentials to authenticate in RGW. Did you enable the Keystone integration in RGW? Also, as far as I know, this admin endpoint needs a RGW admin. I am not sure if the Keystone and RGW integration would enable/make it possible for someone to authenticate as an admin in RGW. Can you check it? To see if you can call that endpoint with these credentials.

On Wed, Sep 28, 2022 at 6:01 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Pollster YML configuration :

---

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: <ACCESS_KEY>,<SECRET_KEY>,<FQDN>/object-store/

  user_id_attribute: "user"

  project_id_attribute: "user"

  resource_id_attribute: "user"

  response_entries_key: "summary"

ACCESS_KEY and SECRET_KEY have been created with “openstack ec2 credentials create”.

Ceilometer central is deployed with OSA and it uses awsauth.py module.

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 02:01 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you show your YML configuration? Also, did you install the AWS authentication module in the container/host where Ceilometer central is running?

On Mon, Sep 26, 2022 at 12:58 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello Rafael,

Thanks for the information about ceilometer patches but for now I’m testing with the credentials in the dynamic pollster config file. I will use barbican when I push all this to production.

The keystone authentication performed by the rados gw with the credentials provided by ceilometer still does not work. I wonder if this could be a S3 signature version issue on ceilometer side, that is on S3 client side. This kind of issue exists with the s3 client “s3cmd” and you have to add “—signature-v2” so that “s3cmd” works well.

What do you think ? Do you know which version of S3 signature ceilometer uses while authenticating ?

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 7 septembre 2022 19:23 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Jean, there are two problems with the Ceilometer. I just opened the patches to resolve it:

• https://review.opendev.org/c/openstack/ceilometer/+/856305

• https://review.opendev.org/c/openstack/ceilometer/+/856304

Without these patches, you might have problems to use Ceilometer with Non-OpenStack dynamic pollsters and barbican credentials.

On Wed, Aug 31, 2022 at 3:55 PM Rafael Weingärtner < rafaelweingartner@gmail.com> wrote:

It is the RGW user that you have. This user must have the role that is needed to access the usage feature in RGW. If I am not mistaken, it required an admin user.

On Wed, Aug 31, 2022 at 1:54 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Thanks to your help, I am close to the goal. Dynamic pollster is loaded and triggered.

But I get a “Status[403] and reason [Forbidden]” in ceilometer logs while requesting admin/usage.

I’m not sure to understand well the auth mechanism. Are we talking about keystone credentials, ec2 credentials, Rados GW user ?...

For now, in testing phase, I use “authentication_parameters”, not barbican.

-JF

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mardi, 30 août 2022 14:17 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Yes, you will need to enable the metric/pollster to be processed. That is done via "polling.yml" file. Also, do not forget that you will need to configure Ceilometer to push this new metric. If you use Gnocchi as the backend, you will need to change/update the gnocchi resource YML file. That file maps resources and metrics in the Gnocchi backend. The configuration resides in Ceilometer. You can create/define new resource types and map them to specific metrics. It depends on how you structure your solution.

P.S. You do not need to use "authentication_parameters". You can use the barbican integration to avoid setting your credentials in a file.

On Tue, Aug 30, 2022 at 9:11 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello,

I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer logs, that it’s actually loaded. But it looks like it was not triggered, I see no trace of ceilometer connection in Rados GW logs.

My definition:

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/swift/v1/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN>

  user_id_attribute: "admin"

  project_id_attribute: "admin"

  resource_id_attribute: "admin"

  response_entries_key: "summary"

Do I have to set an option in ceilometer.conf, or elsewhere, to get my Rados GW dynamic pollster triggered ?

-JF

*From:* Taltavull Jean-François *Sent:* lundi, 29 août 2022 18:41 *To:* 'Rafael Weingärtner' rafaelweingartner@gmail.com *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

Thanks a lot for your quick answer, Rafael !

I will explore this approach.

Jean-Francois

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* lundi, 29 août 2022 17:54 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

You could use a different approach. You can use Dynamic pollster [1], and create your own mechanism to collect data, without needing to change Ceilometer code. Basically all hard-coded pollsters can be converted to a dynamic pollster that is defined in YML.

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollste...

On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hi All,

In our OpenStack deployment, API endpoints are defined by using URLs instead of port numbers and HAProxy forwards requests to the right bakend after having ACLed the URL.

In the case of our object-store service, based on RadosGW, the internal API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>"

When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API with the object-store internal endpoint, the URL becomes https://<FQDN>/admin, as shown by HAProxy logs. This URL does not match any API endpoint from HAProxy point of view. The line of code that rewrites the URL is this one: https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilomete...

What would you think of adding a mechanism based on new Ceilometer configuration option(s) to control the URL rewriting ?

Our deployment characteristics:

• OpenStack release: Wallaby
• Ceph and RadosGW version: 15.2.16
• deployment tool: OSA 23.2.1 and ceph-ansible

Best regards, Jean-Francois

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

Can you also execute the following: ``` python

import awsauth

awsauth ```

That will output a path, and then you can `cat <path>`, example: `cat /var/lib/kolla/venv/lib/python3.8/site-packages/awsauth.py`

On Wed, Sep 28, 2022 at 1:21 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

I removed trailing ‘/object-store/’ from the last value of authentication_parameters

I also:

• disabled s3 keystone auth in RGW

• created a RGW “admin” user with the right privileges to allow admin API

calls

• put RGW in debug mode

And here is what I get in RGW logs:

get_usage string_to_sign=GET Wed, 28 Sep 2022 16:15:45 GMT /admin/usage

get_usage server signature=BlaBlaBlaBla

get_usage client signature=BloBloBlo

get_usage compare=-75

get_usage rgw::auth::s3::LocalEngine denied with reason=-2027

get_usage rgw::auth::s3::AWSAuthStrategy denied with reason=-2027

get_usage rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy

get_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 13:15 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

I think that the last parameter "<FQDN>/object-store/", should be only " <FQDN>". Can you test it?

You are using EC2 credentials to authenticate in RGW. Did you enable the Keystone integration in RGW?

Also, as far as I know, this admin endpoint needs a RGW admin. I am not sure if the Keystone and RGW integration would enable/make it possible for someone to authenticate as an admin in RGW. Can you check it? To see if you can call that endpoint with these credentials.

On Wed, Sep 28, 2022 at 6:01 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Pollster YML configuration :

---

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: <ACCESS_KEY>,<SECRET_KEY>,<FQDN>/object-store/

  user_id_attribute: "user"

  project_id_attribute: "user"

  resource_id_attribute: "user"

  response_entries_key: "summary"

ACCESS_KEY and SECRET_KEY have been created with “openstack ec2 credentials create”.

Ceilometer central is deployed with OSA and it uses awsauth.py module.

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 02:01 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you show your YML configuration? Also, did you install the AWS authentication module in the container/host where Ceilometer central is running?

On Mon, Sep 26, 2022 at 12:58 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello Rafael,

Thanks for the information about ceilometer patches but for now I’m testing with the credentials in the dynamic pollster config file. I will use barbican when I push all this to production.

The keystone authentication performed by the rados gw with the credentials provided by ceilometer still does not work. I wonder if this could be a S3 signature version issue on ceilometer side, that is on S3 client side. This kind of issue exists with the s3 client “s3cmd” and you have to add “—signature-v2” so that “s3cmd” works well.

What do you think ? Do you know which version of S3 signature ceilometer uses while authenticating ?

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 7 septembre 2022 19:23 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Jean, there are two problems with the Ceilometer. I just opened the patches to resolve it:

• https://review.opendev.org/c/openstack/ceilometer/+/856305

• https://review.opendev.org/c/openstack/ceilometer/+/856304

Without these patches, you might have problems to use Ceilometer with Non-OpenStack dynamic pollsters and barbican credentials.

On Wed, Aug 31, 2022 at 3:55 PM Rafael Weingärtner < rafaelweingartner@gmail.com> wrote:

It is the RGW user that you have. This user must have the role that is needed to access the usage feature in RGW. If I am not mistaken, it required an admin user.

On Wed, Aug 31, 2022 at 1:54 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Thanks to your help, I am close to the goal. Dynamic pollster is loaded and triggered.

But I get a “Status[403] and reason [Forbidden]” in ceilometer logs while requesting admin/usage.

I’m not sure to understand well the auth mechanism. Are we talking about keystone credentials, ec2 credentials, Rados GW user ?...

For now, in testing phase, I use “authentication_parameters”, not barbican.

-JF

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mardi, 30 août 2022 14:17 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Yes, you will need to enable the metric/pollster to be processed. That is done via "polling.yml" file. Also, do not forget that you will need to configure Ceilometer to push this new metric. If you use Gnocchi as the backend, you will need to change/update the gnocchi resource YML file. That file maps resources and metrics in the Gnocchi backend. The configuration resides in Ceilometer. You can create/define new resource types and map them to specific metrics. It depends on how you structure your solution.

P.S. You do not need to use "authentication_parameters". You can use the barbican integration to avoid setting your credentials in a file.

On Tue, Aug 30, 2022 at 9:11 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello,

I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer logs, that it’s actually loaded. But it looks like it was not triggered, I see no trace of ceilometer connection in Rados GW logs.

My definition:

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/swift/v1/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN>

  user_id_attribute: "admin"

  project_id_attribute: "admin"

  resource_id_attribute: "admin"

  response_entries_key: "summary"

Do I have to set an option in ceilometer.conf, or elsewhere, to get my Rados GW dynamic pollster triggered ?

-JF

*From:* Taltavull Jean-François *Sent:* lundi, 29 août 2022 18:41 *To:* 'Rafael Weingärtner' rafaelweingartner@gmail.com *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

Thanks a lot for your quick answer, Rafael !

I will explore this approach.

Jean-Francois

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* lundi, 29 août 2022 17:54 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

You could use a different approach. You can use Dynamic pollster [1], and create your own mechanism to collect data, without needing to change Ceilometer code. Basically all hard-coded pollsters can be converted to a dynamic pollster that is defined in YML.

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollste...

On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hi All,

In our OpenStack deployment, API endpoints are defined by using URLs instead of port numbers and HAProxy forwards requests to the right bakend after having ACLed the URL.

In the case of our object-store service, based on RadosGW, the internal API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>"

When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API with the object-store internal endpoint, the URL becomes https://<FQDN>/admin, as shown by HAProxy logs. This URL does not match any API endpoint from HAProxy point of view. The line of code that rewrites the URL is this one: https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilomete...

What would you think of adding a mechanism based on new Ceilometer configuration option(s) to control the URL rewriting ?

Our deployment characteristics:

• OpenStack release: Wallaby
• Ceph and RadosGW version: 15.2.16
• deployment tool: OSA 23.2.1 and ceph-ansible

Best regards, Jean-Francois

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

Can you test you credentials with the following code?

```

import json import requests import os

import six.moves.urllib.parse as urlparse

class RGWAdminAPIFailed(Exception): pass

if __name__ == '__main__':

rados_gw_base_url = "put your RGW URL here. E.g. http://server.com:port /something" print("Executing test on: [%s]." % rados_gw_base_url)

rados_gw_admin_context = "/admin"

rados_gw_path = "/usage?stats=True"

print("Rados GW admin context [%s] and path [%s] used." % (rados_gw_admin_context, rados_gw_path))

rados_gw_request_url = urlparse.urljoin(rados_gw_base_url, '/admin') + '/bucket?stats=True' print("Rados GW request URL [%s]." % rados_gw_request_url)

rados_gw_access_key_to_use = "put your access key here" rados_gw_secret_key_to_use = "put your secret key here"

rados_gw_host_name = urlparse.urlparse(rados_gw_request_url).netloc print("Rados GW host: %s" % rados_gw_host_name) module_name = "awsauth" class_name = "S3Auth" arguments = [rados_gw_access_key_to_use, rados_gw_secret_key_to_use, rados_gw_host_name] module = __import__(module_name) class_ = getattr(module, class_name) instance = class_(*arguments)

r = requests.get( rados_gw_request_url, auth=instance, timeout=30) #auth=awsauth.S3Auth(*arguments))

if r.status_code != 200: raise RGWAdminAPIFailed( ('RGW AdminOps API returned %(status)s %(reason)s') % {'status': r.status_code, 'reason': r.reason})

response_body = r.text parsed_json = json.loads(response_body)

print("Response cookies: [%s]." % r.cookies)

radosGw_output_file = "/home/<user_here>/Downloads/radosGw-usage.json"

if os.path.exists(radosGw_output_file): os.remove(radosGw_output_file)

with open(radosGw_output_file, "w") as file1: file1.writelines(json.dumps(parsed_json, indent=4, sort_keys=True)) file1.flush()

exit(0)

```

On Thu, Sep 29, 2022 at 4:09 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

python

Python 3.8.10 (default, Sep 28 2021, 16:10:42)

[GCC 9.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

...

...

...

import awsauth

...

...

...

awsauth

<module 'awsauth' from '/openstack/venvs/ceilometer-23.2.0/lib/python3.8/site-packages/awsauth.py'>

...

...

...

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 18:40 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you also execute the following:

python



import awsauth



awsauth

That will output a path, and then you can `cat <path>`, example: `cat /var/lib/kolla/venv/lib/python3.8/site-packages/awsauth.py`

On Wed, Sep 28, 2022 at 1:21 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

I removed trailing ‘/object-store/’ from the last value of authentication_parameters

I also:

• disabled s3 keystone auth in RGW

• created a RGW “admin” user with the right privileges to allow admin API

calls

• put RGW in debug mode

And here is what I get in RGW logs:

get_usage string_to_sign=GET Wed, 28 Sep 2022 16:15:45 GMT /admin/usage

get_usage server signature=BlaBlaBlaBla

get_usage client signature=BloBloBlo

get_usage compare=-75

get_usage rgw::auth::s3::LocalEngine denied with reason=-2027

get_usage rgw::auth::s3::AWSAuthStrategy denied with reason=-2027

get_usage rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy

get_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 13:15 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

I think that the last parameter "<FQDN>/object-store/", should be only " <FQDN>". Can you test it?

You are using EC2 credentials to authenticate in RGW. Did you enable the Keystone integration in RGW?

Also, as far as I know, this admin endpoint needs a RGW admin. I am not sure if the Keystone and RGW integration would enable/make it possible for someone to authenticate as an admin in RGW. Can you check it? To see if you can call that endpoint with these credentials.

On Wed, Sep 28, 2022 at 6:01 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Pollster YML configuration :

---

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: <ACCESS_KEY>,<SECRET_KEY>,<FQDN>/object-store/

  user_id_attribute: "user"

  project_id_attribute: "user"

  resource_id_attribute: "user"

  response_entries_key: "summary"

ACCESS_KEY and SECRET_KEY have been created with “openstack ec2 credentials create”.

Ceilometer central is deployed with OSA and it uses awsauth.py module.

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 02:01 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you show your YML configuration? Also, did you install the AWS authentication module in the container/host where Ceilometer central is running?

On Mon, Sep 26, 2022 at 12:58 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello Rafael,

Thanks for the information about ceilometer patches but for now I’m testing with the credentials in the dynamic pollster config file. I will use barbican when I push all this to production.

The keystone authentication performed by the rados gw with the credentials provided by ceilometer still does not work. I wonder if this could be a S3 signature version issue on ceilometer side, that is on S3 client side. This kind of issue exists with the s3 client “s3cmd” and you have to add “—signature-v2” so that “s3cmd” works well.

What do you think ? Do you know which version of S3 signature ceilometer uses while authenticating ?

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 7 septembre 2022 19:23 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Jean, there are two problems with the Ceilometer. I just opened the patches to resolve it:

• https://review.opendev.org/c/openstack/ceilometer/+/856305

• https://review.opendev.org/c/openstack/ceilometer/+/856304

Without these patches, you might have problems to use Ceilometer with Non-OpenStack dynamic pollsters and barbican credentials.

On Wed, Aug 31, 2022 at 3:55 PM Rafael Weingärtner < rafaelweingartner@gmail.com> wrote:

It is the RGW user that you have. This user must have the role that is needed to access the usage feature in RGW. If I am not mistaken, it required an admin user.

On Wed, Aug 31, 2022 at 1:54 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Thanks to your help, I am close to the goal. Dynamic pollster is loaded and triggered.

But I get a “Status[403] and reason [Forbidden]” in ceilometer logs while requesting admin/usage.

I’m not sure to understand well the auth mechanism. Are we talking about keystone credentials, ec2 credentials, Rados GW user ?...

For now, in testing phase, I use “authentication_parameters”, not barbican.

-JF

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mardi, 30 août 2022 14:17 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Yes, you will need to enable the metric/pollster to be processed. That is done via "polling.yml" file. Also, do not forget that you will need to configure Ceilometer to push this new metric. If you use Gnocchi as the backend, you will need to change/update the gnocchi resource YML file. That file maps resources and metrics in the Gnocchi backend. The configuration resides in Ceilometer. You can create/define new resource types and map them to specific metrics. It depends on how you structure your solution.

P.S. You do not need to use "authentication_parameters". You can use the barbican integration to avoid setting your credentials in a file.

On Tue, Aug 30, 2022 at 9:11 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello,

I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer logs, that it’s actually loaded. But it looks like it was not triggered, I see no trace of ceilometer connection in Rados GW logs.

My definition:

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/swift/v1/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN>

  user_id_attribute: "admin"

  project_id_attribute: "admin"

  resource_id_attribute: "admin"

  response_entries_key: "summary"

Do I have to set an option in ceilometer.conf, or elsewhere, to get my Rados GW dynamic pollster triggered ?

-JF

*From:* Taltavull Jean-François *Sent:* lundi, 29 août 2022 18:41 *To:* 'Rafael Weingärtner' rafaelweingartner@gmail.com *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

Thanks a lot for your quick answer, Rafael !

I will explore this approach.

Jean-Francois

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* lundi, 29 août 2022 17:54 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

You could use a different approach. You can use Dynamic pollster [1], and create your own mechanism to collect data, without needing to change Ceilometer code. Basically all hard-coded pollsters can be converted to a dynamic pollster that is defined in YML.

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollste...

On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hi All,

In our OpenStack deployment, API endpoints are defined by using URLs instead of port numbers and HAProxy forwards requests to the right bakend after having ACLed the URL.

In the case of our object-store service, based on RadosGW, the internal API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>"

When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API with the object-store internal endpoint, the URL becomes https://<FQDN>/admin, as shown by HAProxy logs. This URL does not match any API endpoint from HAProxy point of view. The line of code that rewrites the URL is this one: https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilomete...

What would you think of adding a mechanism based on new Ceilometer configuration option(s) to control the URL rewriting ?

Our deployment characteristics:

• OpenStack release: Wallaby
• Ceph and RadosGW version: 15.2.16
• deployment tool: OSA 23.2.1 and ceph-ansible

Best regards, Jean-Francois

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

This is the signature used by the `awsauth` library: ```

def get_signature(self, r): canonical_string = self.get_canonical_string( r.url, r.headers, r.method) if py3k: key = self.secret_key.encode('utf-8') msg = canonical_string.encode('utf-8') else: key = self.secret_key msg = canonical_string h = hmac.new(key, msg, digestmod=sha) return encodestring(h.digest()).strip()

```

After that is generated, it is added in the headers:

# Create date header if it is not created yet. if 'date' not in r.headers and 'x-amz-date' not in r.headers: r.headers['date'] = formatdate( timeval=None, localtime=False, usegmt=True) signature = self.get_signature(r) if py3k: signature = signature.decode('utf-8') r.headers['Authorization'] = 'AWS %s:%s' % (self.access_key, signature)

On Thu, Sep 29, 2022 at 9:15 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

$ python test_creds.py

Executing test on: [FQDN/object-store/].

Rados GW admin context [/admin] and path [/usage?stats=True] used.

Rados GW request URL [http://FQDN/object-store/admin/bucket?stats=True].

Rados GW host: FQDN

Traceback (most recent call last):

  File "test_creds.py", line 45, in <module>

    raise RGWAdminAPIFailed(

__main__.RGWAdminAPIFailed: RGW AdminOps API returned 403 Forbidden

So the same as with ceilometer. Auth is done by RGW, not by keystone, and the ceph “admin” user exists and owns the right privileges:

$ sudo radosgw-admin user info --uid
admin
                                                                                   [22/296]{

    "user_id": "admin",

    "display_name": "admin user",

    "email": "",

    "suspended": 0,

    "max_buckets": 1000,

    "subusers": [],

    "keys": [

        {

            "user": "admin",

            "access_key": “admin_access_key",

            "secret_key": "admin_secret_key"

        }

    ],

    "swift_keys": [],

    "caps": [

        {

            "type": "buckets",

            "perm": "*"

        },

        {

            "type": "metadata",

            "perm": "*"

        },


{
                                                                                                                                                                          "type":
"usage",
                                                                                                                                                                                         "perm":
"*"
                                                                                                                                                                                                    },
                                                                                                                                                                                                                   {

            "type": "users",
                                                                                                                                                                                                                                                "perm":
"*"
                                                                                                                                                                                                                                                           }
                                                                                                                                                                                                                                                                      ],

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* jeudi, 29 septembre 2022 12:32 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you test you credentials with the following code?

import json
import requests
import os

import six.moves.urllib.parse as urlparse


class RGWAdminAPIFailed(Exception):
    pass


if __name__ == '__main__':

    rados_gw_base_url = "put your RGW URL here. E.g.
http://server.com:port/something"
    print("Executing test on: [%s]." % rados_gw_base_url)

    rados_gw_admin_context = "/admin"

    rados_gw_path = "/usage?stats=True"

    print("Rados GW admin context [%s] and path [%s] used." %
(rados_gw_admin_context, rados_gw_path))

    rados_gw_request_url = urlparse.urljoin(rados_gw_base_url, '/admin') +
'/bucket?stats=True'
    print("Rados GW request URL [%s]." % rados_gw_request_url)

    rados_gw_access_key_to_use = "put your access key here"
    rados_gw_secret_key_to_use = "put your secret key here"

    rados_gw_host_name = urlparse.urlparse(rados_gw_request_url).netloc
    print("Rados GW host: %s" % rados_gw_host_name)
    module_name = "awsauth"
    class_name = "S3Auth"
    arguments = [rados_gw_access_key_to_use, rados_gw_secret_key_to_use,
rados_gw_host_name]
    module = __import__(module_name)
    class_ = getattr(module, class_name)
    instance = class_(*arguments)

    r = requests.get(
        rados_gw_request_url,
        auth=instance, timeout=30)
        #auth=awsauth.S3Auth(*arguments))


    if r.status_code != 200:
        raise RGWAdminAPIFailed(
            ('RGW AdminOps API returned %(status)s %(reason)s') %
            {'status': r.status_code, 'reason': r.reason})

    response_body = r.text
    parsed_json = json.loads(response_body)

    print("Response cookies: [%s]." % r.cookies)

    radosGw_output_file = "/home/<user_here>/Downloads/radosGw-usage.json"

    if os.path.exists(radosGw_output_file):
        os.remove(radosGw_output_file)

    with open(radosGw_output_file, "w") as file1:
        file1.writelines(json.dumps(parsed_json, indent=4, sort_keys=True))
        file1.flush()

    exit(0)

On Thu, Sep 29, 2022 at 4:09 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

python

Python 3.8.10 (default, Sep 28 2021, 16:10:42)

[GCC 9.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

...

...

...

import awsauth

...

...

...

awsauth

<module 'awsauth' from '/openstack/venvs/ceilometer-23.2.0/lib/python3.8/site-packages/awsauth.py'>

...

...

...

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 18:40 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you also execute the following:

python



import awsauth



awsauth

That will output a path, and then you can `cat <path>`, example: `cat /var/lib/kolla/venv/lib/python3.8/site-packages/awsauth.py`

On Wed, Sep 28, 2022 at 1:21 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

I removed trailing ‘/object-store/’ from the last value of authentication_parameters

I also:

• disabled s3 keystone auth in RGW

• created a RGW “admin” user with the right privileges to allow admin API

calls

• put RGW in debug mode

And here is what I get in RGW logs:

get_usage string_to_sign=GET Wed, 28 Sep 2022 16:15:45 GMT /admin/usage

get_usage server signature=BlaBlaBlaBla

get_usage client signature=BloBloBlo

get_usage compare=-75

get_usage rgw::auth::s3::LocalEngine denied with reason=-2027

get_usage rgw::auth::s3::AWSAuthStrategy denied with reason=-2027

get_usage rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy

get_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 13:15 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

I think that the last parameter "<FQDN>/object-store/", should be only " <FQDN>". Can you test it?

You are using EC2 credentials to authenticate in RGW. Did you enable the Keystone integration in RGW?

Also, as far as I know, this admin endpoint needs a RGW admin. I am not sure if the Keystone and RGW integration would enable/make it possible for someone to authenticate as an admin in RGW. Can you check it? To see if you can call that endpoint with these credentials.

On Wed, Sep 28, 2022 at 6:01 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Pollster YML configuration :

---

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: <ACCESS_KEY>,<SECRET_KEY>,<FQDN>/object-store/

  user_id_attribute: "user"

  project_id_attribute: "user"

  resource_id_attribute: "user"

  response_entries_key: "summary"

ACCESS_KEY and SECRET_KEY have been created with “openstack ec2 credentials create”.

Ceilometer central is deployed with OSA and it uses awsauth.py module.

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 02:01 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you show your YML configuration? Also, did you install the AWS authentication module in the container/host where Ceilometer central is running?

On Mon, Sep 26, 2022 at 12:58 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello Rafael,

Thanks for the information about ceilometer patches but for now I’m testing with the credentials in the dynamic pollster config file. I will use barbican when I push all this to production.

The keystone authentication performed by the rados gw with the credentials provided by ceilometer still does not work. I wonder if this could be a S3 signature version issue on ceilometer side, that is on S3 client side. This kind of issue exists with the s3 client “s3cmd” and you have to add “—signature-v2” so that “s3cmd” works well.

What do you think ? Do you know which version of S3 signature ceilometer uses while authenticating ?

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 7 septembre 2022 19:23 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Jean, there are two problems with the Ceilometer. I just opened the patches to resolve it:

• https://review.opendev.org/c/openstack/ceilometer/+/856305

• https://review.opendev.org/c/openstack/ceilometer/+/856304

Without these patches, you might have problems to use Ceilometer with Non-OpenStack dynamic pollsters and barbican credentials.

On Wed, Aug 31, 2022 at 3:55 PM Rafael Weingärtner < rafaelweingartner@gmail.com> wrote:

It is the RGW user that you have. This user must have the role that is needed to access the usage feature in RGW. If I am not mistaken, it required an admin user.

On Wed, Aug 31, 2022 at 1:54 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Thanks to your help, I am close to the goal. Dynamic pollster is loaded and triggered.

But I get a “Status[403] and reason [Forbidden]” in ceilometer logs while requesting admin/usage.

I’m not sure to understand well the auth mechanism. Are we talking about keystone credentials, ec2 credentials, Rados GW user ?...

For now, in testing phase, I use “authentication_parameters”, not barbican.

-JF

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mardi, 30 août 2022 14:17 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Yes, you will need to enable the metric/pollster to be processed. That is done via "polling.yml" file. Also, do not forget that you will need to configure Ceilometer to push this new metric. If you use Gnocchi as the backend, you will need to change/update the gnocchi resource YML file. That file maps resources and metrics in the Gnocchi backend. The configuration resides in Ceilometer. You can create/define new resource types and map them to specific metrics. It depends on how you structure your solution.

P.S. You do not need to use "authentication_parameters". You can use the barbican integration to avoid setting your credentials in a file.

On Tue, Aug 30, 2022 at 9:11 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello,

I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer logs, that it’s actually loaded. But it looks like it was not triggered, I see no trace of ceilometer connection in Rados GW logs.

My definition:

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/swift/v1/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN>

  user_id_attribute: "admin"

  project_id_attribute: "admin"

  resource_id_attribute: "admin"

  response_entries_key: "summary"

Do I have to set an option in ceilometer.conf, or elsewhere, to get my Rados GW dynamic pollster triggered ?

-JF

*From:* Taltavull Jean-François *Sent:* lundi, 29 août 2022 18:41 *To:* 'Rafael Weingärtner' rafaelweingartner@gmail.com *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

Thanks a lot for your quick answer, Rafael !

I will explore this approach.

Jean-Francois

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* lundi, 29 août 2022 17:54 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

You could use a different approach. You can use Dynamic pollster [1], and create your own mechanism to collect data, without needing to change Ceilometer code. Basically all hard-coded pollsters can be converted to a dynamic pollster that is defined in YML.

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollste...

On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hi All,

In our OpenStack deployment, API endpoints are defined by using URLs instead of port numbers and HAProxy forwards requests to the right bakend after having ACLed the URL.

In the case of our object-store service, based on RadosGW, the internal API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>"

When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API with the object-store internal endpoint, the URL becomes https://<FQDN>/admin, as shown by HAProxy logs. This URL does not match any API endpoint from HAProxy point of view. The line of code that rewrites the URL is this one: https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilomete...

What would you think of adding a mechanism based on new Ceilometer configuration option(s) to control the URL rewriting ?

Our deployment characteristics:

• OpenStack release: Wallaby
• Ceph and RadosGW version: 15.2.16
• deployment tool: OSA 23.2.1 and ceph-ansible

Best regards, Jean-Francois

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

No, I just showed you the code, so you can see how the authentication is being executed, and where/how the parameters are set in the headers. It is a bit odd, I have used this so many times, and it always works. What is your RGW instance version?

On Fri, Sep 30, 2022 at 4:09 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Do you mean the issue comes from how the `awsauth` module handles the signature ?

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* jeudi, 29 septembre 2022 17:23 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

This is the signature used by the `awsauth` library:

def get_signature(self, r):
    canonical_string = self.get_canonical_string(
        r.url, r.headers, r.method)
    if py3k:
        key = self.secret_key.encode('utf-8')
        msg = canonical_string.encode('utf-8')
    else:
        key = self.secret_key
        msg = canonical_string
    h = hmac.new(key, msg, digestmod=sha)
    return encodestring(h.digest()).strip()

After that is generated, it is added in the headers:

# Create date header if it is not created yet. if 'date' not in r.headers and 'x-amz-date' not in r.headers: r.headers['date'] = formatdate( timeval=None, localtime=False, usegmt=True) signature = self.get_signature(r) if py3k: signature = signature.decode('utf-8') r.headers['Authorization'] = 'AWS %s:%s' % (self.access_key, signature)

On Thu, Sep 29, 2022 at 9:15 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

$ python test_creds.py

Executing test on: [FQDN/object-store/].

Rados GW admin context [/admin] and path [/usage?stats=True] used.

Rados GW request URL [http://FQDN/object-store/admin/bucket?stats=True].

Rados GW host: FQDN

Traceback (most recent call last):

  File "test_creds.py", line 45, in <module>

    raise RGWAdminAPIFailed(

__main__.RGWAdminAPIFailed: RGW AdminOps API returned 403 Forbidden

So the same as with ceilometer. Auth is done by RGW, not by keystone, and the ceph “admin” user exists and owns the right privileges:

$ sudo radosgw-admin user info --uid
admin
                                                                                   [22/296]{

    "user_id": "admin",

    "display_name": "admin user",

    "email": "",

    "suspended": 0,

    "max_buckets": 1000,

    "subusers": [],

    "keys": [

        {

            "user": "admin",

            "access_key": “admin_access_key",

            "secret_key": "admin_secret_key"

        }

    ],

    "swift_keys": [],

    "caps": [

        {

            "type": "buckets",

            "perm": "*"

        },

        {

            "type": "metadata",

            "perm": "*"

        },


{
                                                                                                                                                                          "type":
"usage",
                                                                                                                                                                                         "perm":
"*"
                                                                                                                                                                                                    },
                                                                                                                                                                                                                   {

            "type": "users",
                                                                                                                                                                                                                                                "perm":
"*"
                                                                                                                                                                                                                                                           }
                                                                                                                                                                                                                                                                      ],

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* jeudi, 29 septembre 2022 12:32 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you test you credentials with the following code?

import json
import requests
import os

import six.moves.urllib.parse as urlparse


class RGWAdminAPIFailed(Exception):
    pass


if __name__ == '__main__':

    rados_gw_base_url = "put your RGW URL here. E.g.
http://server.com:port/something"
    print("Executing test on: [%s]." % rados_gw_base_url)

    rados_gw_admin_context = "/admin"

    rados_gw_path = "/usage?stats=True"

    print("Rados GW admin context [%s] and path [%s] used." %
(rados_gw_admin_context, rados_gw_path))

    rados_gw_request_url = urlparse.urljoin(rados_gw_base_url, '/admin') +
'/bucket?stats=True'
    print("Rados GW request URL [%s]." % rados_gw_request_url)

    rados_gw_access_key_to_use = "put your access key here"
    rados_gw_secret_key_to_use = "put your secret key here"

    rados_gw_host_name = urlparse.urlparse(rados_gw_request_url).netloc
    print("Rados GW host: %s" % rados_gw_host_name)
    module_name = "awsauth"
    class_name = "S3Auth"
    arguments = [rados_gw_access_key_to_use, rados_gw_secret_key_to_use,
rados_gw_host_name]
    module = __import__(module_name)
    class_ = getattr(module, class_name)
    instance = class_(*arguments)

    r = requests.get(
        rados_gw_request_url,
        auth=instance, timeout=30)
        #auth=awsauth.S3Auth(*arguments))


    if r.status_code != 200:
        raise RGWAdminAPIFailed(
            ('RGW AdminOps API returned %(status)s %(reason)s') %
            {'status': r.status_code, 'reason': r.reason})

    response_body = r.text
    parsed_json = json.loads(response_body)

    print("Response cookies: [%s]." % r.cookies)

    radosGw_output_file = "/home/<user_here>/Downloads/radosGw-usage.json"

    if os.path.exists(radosGw_output_file):
        os.remove(radosGw_output_file)

    with open(radosGw_output_file, "w") as file1:
        file1.writelines(json.dumps(parsed_json, indent=4, sort_keys=True))
        file1.flush()

    exit(0)

On Thu, Sep 29, 2022 at 4:09 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

python

Python 3.8.10 (default, Sep 28 2021, 16:10:42)

[GCC 9.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

...

...

...

import awsauth

...

...

...

awsauth

<module 'awsauth' from '/openstack/venvs/ceilometer-23.2.0/lib/python3.8/site-packages/awsauth.py'>

...

...

...

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 18:40 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you also execute the following:

python



import awsauth



awsauth

That will output a path, and then you can `cat <path>`, example: `cat /var/lib/kolla/venv/lib/python3.8/site-packages/awsauth.py`

On Wed, Sep 28, 2022 at 1:21 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

I removed trailing ‘/object-store/’ from the last value of authentication_parameters

I also:

• disabled s3 keystone auth in RGW

• created a RGW “admin” user with the right privileges to allow admin API

calls

• put RGW in debug mode

And here is what I get in RGW logs:

get_usage string_to_sign=GET Wed, 28 Sep 2022 16:15:45 GMT /admin/usage

get_usage server signature=BlaBlaBlaBla

get_usage client signature=BloBloBlo

get_usage compare=-75

get_usage rgw::auth::s3::LocalEngine denied with reason=-2027

get_usage rgw::auth::s3::AWSAuthStrategy denied with reason=-2027

get_usage rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy

get_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 13:15 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

I think that the last parameter "<FQDN>/object-store/", should be only " <FQDN>". Can you test it?

You are using EC2 credentials to authenticate in RGW. Did you enable the Keystone integration in RGW?

Also, as far as I know, this admin endpoint needs a RGW admin. I am not sure if the Keystone and RGW integration would enable/make it possible for someone to authenticate as an admin in RGW. Can you check it? To see if you can call that endpoint with these credentials.

On Wed, Sep 28, 2022 at 6:01 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Pollster YML configuration :

---

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: <ACCESS_KEY>,<SECRET_KEY>,<FQDN>/object-store/

  user_id_attribute: "user"

  project_id_attribute: "user"

  resource_id_attribute: "user"

  response_entries_key: "summary"

ACCESS_KEY and SECRET_KEY have been created with “openstack ec2 credentials create”.

Ceilometer central is deployed with OSA and it uses awsauth.py module.

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 02:01 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you show your YML configuration? Also, did you install the AWS authentication module in the container/host where Ceilometer central is running?

On Mon, Sep 26, 2022 at 12:58 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello Rafael,

Thanks for the information about ceilometer patches but for now I’m testing with the credentials in the dynamic pollster config file. I will use barbican when I push all this to production.

The keystone authentication performed by the rados gw with the credentials provided by ceilometer still does not work. I wonder if this could be a S3 signature version issue on ceilometer side, that is on S3 client side. This kind of issue exists with the s3 client “s3cmd” and you have to add “—signature-v2” so that “s3cmd” works well.

What do you think ? Do you know which version of S3 signature ceilometer uses while authenticating ?

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 7 septembre 2022 19:23 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Jean, there are two problems with the Ceilometer. I just opened the patches to resolve it:

• https://review.opendev.org/c/openstack/ceilometer/+/856305

• https://review.opendev.org/c/openstack/ceilometer/+/856304

Without these patches, you might have problems to use Ceilometer with Non-OpenStack dynamic pollsters and barbican credentials.

On Wed, Aug 31, 2022 at 3:55 PM Rafael Weingärtner < rafaelweingartner@gmail.com> wrote:

It is the RGW user that you have. This user must have the role that is needed to access the usage feature in RGW. If I am not mistaken, it required an admin user.

On Wed, Aug 31, 2022 at 1:54 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Thanks to your help, I am close to the goal. Dynamic pollster is loaded and triggered.

But I get a “Status[403] and reason [Forbidden]” in ceilometer logs while requesting admin/usage.

I’m not sure to understand well the auth mechanism. Are we talking about keystone credentials, ec2 credentials, Rados GW user ?...

For now, in testing phase, I use “authentication_parameters”, not barbican.

-JF

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mardi, 30 août 2022 14:17 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Yes, you will need to enable the metric/pollster to be processed. That is done via "polling.yml" file. Also, do not forget that you will need to configure Ceilometer to push this new metric. If you use Gnocchi as the backend, you will need to change/update the gnocchi resource YML file. That file maps resources and metrics in the Gnocchi backend. The configuration resides in Ceilometer. You can create/define new resource types and map them to specific metrics. It depends on how you structure your solution.

P.S. You do not need to use "authentication_parameters". You can use the barbican integration to avoid setting your credentials in a file.

On Tue, Aug 30, 2022 at 9:11 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello,

I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer logs, that it’s actually loaded. But it looks like it was not triggered, I see no trace of ceilometer connection in Rados GW logs.

My definition:

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/swift/v1/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN>

  user_id_attribute: "admin"

  project_id_attribute: "admin"

  resource_id_attribute: "admin"

  response_entries_key: "summary"

Do I have to set an option in ceilometer.conf, or elsewhere, to get my Rados GW dynamic pollster triggered ?

-JF

*From:* Taltavull Jean-François *Sent:* lundi, 29 août 2022 18:41 *To:* 'Rafael Weingärtner' rafaelweingartner@gmail.com *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

Thanks a lot for your quick answer, Rafael !

I will explore this approach.

Jean-Francois

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* lundi, 29 août 2022 17:54 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

You could use a different approach. You can use Dynamic pollster [1], and create your own mechanism to collect data, without needing to change Ceilometer code. Basically all hard-coded pollsters can be converted to a dynamic pollster that is defined in YML.

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollste...

On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hi All,

In our OpenStack deployment, API endpoints are defined by using URLs instead of port numbers and HAProxy forwards requests to the right bakend after having ACLed the URL.

In the case of our object-store service, based on RadosGW, the internal API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>"

When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API with the object-store internal endpoint, the URL becomes https://<FQDN>/admin, as shown by HAProxy logs. This URL does not match any API endpoint from HAProxy point of view. The line of code that rewrites the URL is this one: https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilomete...

What would you think of adding a mechanism based on new Ceilometer configuration option(s) to control the URL rewriting ?

Our deployment characteristics:

• OpenStack release: Wallaby
• Ceph and RadosGW version: 15.2.16
• deployment tool: OSA 23.2.1 and ceph-ansible

Best regards, Jean-Francois

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

Hello Raphaël,

I restored the RGW keystone authentication and did some more tests. The problem is that the S3 request signature provided by ceilometer and the one computed by keystone mismatch.

OpenStack release is Wallaby.

keystone/api/s3tokens.py: ```` class S3Resource(EC2_S3_Resource.ResourceBase): @staticmethod def _check_signature(creds_ref, credentials): string_to_sign = base64.urlsafe_b64decode(str(credentials['token']))

if string_to_sign[0:4] != b'AWS4': signature = _calculate_signature_v1(string_to_sign, creds_ref['secret']) else: signature = _calculate_signature_v4(string_to_sign, creds_ref['secret']) if not utils.auth_str_equal(credentials['signature'], signature): raise exception.Unauthorized( <<<------------------------------------------we fall there message=_('Credential signature mismatch')) ````

From: Taltavull Jean-François Sent: vendredi, 30 septembre 2022 14:48 To: 'Rafael Weingärtner' rafaelweingartner@gmail.com Cc: openstack-discuss openstack-discuss@lists.openstack.org Subject: RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

``` $ sudo /usr/bin/radosgw --version ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable) ```

From: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Sent: vendredi, 30 septembre 2022 12:37 To: Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

No, I just showed you the code, so you can see how the authentication is being executed, and where/how the parameters are set in the headers. It is a bit odd, I have used this so many times, and it always works. What is your RGW instance version?

On Fri, Sep 30, 2022 at 4:09 AM Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> wrote: Do you mean the issue comes from how the `awsauth` module handles the signature ?

From: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Sent: jeudi, 29 septembre 2022 17:23 To: Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

This is the signature used by the `awsauth` library: ```

def get_signature(self, r): canonical_string = self.get_canonical_string( r.url, r.headers, r.method) if py3k: key = self.secret_key.encode('utf-8') msg = canonical_string.encode('utf-8') else: key = self.secret_key msg = canonical_string h = hmac.new(key, msg, digestmod=sha) return encodestring(h.digest()).strip()

```

After that is generated, it is added in the headers:

# Create date header if it is not created yet. if 'date' not in r.headers and 'x-amz-date' not in r.headers: r.headers['date'] = formatdate( timeval=None, localtime=False, usegmt=True) signature = self.get_signature(r) if py3k: signature = signature.decode('utf-8') r.headers['Authorization'] = 'AWS %s:%s' % (self.access_key, signature)

On Thu, Sep 29, 2022 at 9:15 AM Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> wrote: ``` $ python test_creds.py Executing test on: [FQDN/object-store/]. Rados GW admin context [/admin] and path [/usage?stats=True] used. Rados GW request URL [http://FQDN/object-store/admin/bucket?stats=True]. Rados GW host: FQDN Traceback (most recent call last): File "test_creds.py", line 45, in <module> raise RGWAdminAPIFailed( __main__.RGWAdminAPIFailed: RGW AdminOps API returned 403 Forbidden ```

So the same as with ceilometer. Auth is done by RGW, not by keystone, and the ceph “admin” user exists and owns the right privileges: ``` $ sudo radosgw-admin user info --uid admin [22/296]{ "user_id": "admin", "display_name": "admin user", "email": "", "suspended": 0, "max_buckets": 1000, "subusers": [], "keys": [ { "user": "admin", "access_key": “admin_access_key", "secret_key": "admin_secret_key" } ], "swift_keys": [], "caps": [ { "type": "buckets", "perm": "*" }, { "type": "metadata", "perm": "*" }, { "type": "usage", "perm": "*" }, { "type": "users", "perm": "*" } ],

```

From: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Sent: jeudi, 29 septembre 2022 12:32 To: Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

Can you test you credentials with the following code?

```

import json import requests import os

import six.moves.urllib.parse as urlparse

class RGWAdminAPIFailed(Exception): pass

if __name__ == '__main__':

rados_gw_base_url = "put your RGW URL here. E.g. http://server.com:port/something" print("Executing test on: [%s]." % rados_gw_base_url)

rados_gw_admin_context = "/admin"

rados_gw_path = "/usage?stats=True"

print("Rados GW admin context [%s] and path [%s] used." % (rados_gw_admin_context, rados_gw_path))

rados_gw_request_url = urlparse.urljoin(rados_gw_base_url, '/admin') + '/bucket?stats=True' print("Rados GW request URL [%s]." % rados_gw_request_url)

rados_gw_access_key_to_use = "put your access key here" rados_gw_secret_key_to_use = "put your secret key here"

rados_gw_host_name = urlparse.urlparse(rados_gw_request_url).netloc print("Rados GW host: %s" % rados_gw_host_name) module_name = "awsauth" class_name = "S3Auth" arguments = [rados_gw_access_key_to_use, rados_gw_secret_key_to_use, rados_gw_host_name] module = __import__(module_name) class_ = getattr(module, class_name) instance = class_(*arguments)

r = requests.get( rados_gw_request_url, auth=instance, timeout=30) #auth=awsauth.S3Auth(*arguments))

if r.status_code != 200: raise RGWAdminAPIFailed( ('RGW AdminOps API returned %(status)s %(reason)s') % {'status': r.status_code, 'reason': r.reason})

response_body = r.text parsed_json = json.loads(response_body)

print("Response cookies: [%s]." % r.cookies)

radosGw_output_file = "/home/<user_here>/Downloads/radosGw-usage.json"

if os.path.exists(radosGw_output_file): os.remove(radosGw_output_file)

with open(radosGw_output_file, "w") as file1: file1.writelines(json.dumps(parsed_json, indent=4, sort_keys=True)) file1.flush()

exit(0)

```

On Thu, Sep 29, 2022 at 4:09 AM Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> wrote: python Python 3.8.10 (default, Sep 28 2021, 16:10:42) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information.

...

...

import awsauth awsauth

<module 'awsauth' from '/openstack/venvs/ceilometer-23.2.0/lib/python3.8/site-packages/awsauth.py'>

...

...

From: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Sent: mercredi, 28 septembre 2022 18:40 To: Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

Can you also execute the following: ``` python

import awsauth

awsauth ``` That will output a path, and then you can `cat <path>`, example: `cat /var/lib/kolla/venv/lib/python3.8/site-packages/awsauth.py`

On Wed, Sep 28, 2022 at 1:21 PM Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> wrote: I removed trailing ‘/object-store/’ from the last value of authentication_parameters

I also: - disabled s3 keystone auth in RGW - created a RGW “admin” user with the right privileges to allow admin API calls - put RGW in debug mode

And here is what I get in RGW logs:

get_usage string_to_sign=GET Wed, 28 Sep 2022 16:15:45 GMT /admin/usage get_usage server signature=BlaBlaBlaBla get_usage client signature=BloBloBlo get_usage compare=-75 get_usage rgw::auth::s3::LocalEngine denied with reason=-2027 get_usage rgw::auth::s3::AWSAuthStrategy denied with reason=-2027 get_usage rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy get_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine

From: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Sent: mercredi, 28 septembre 2022 13:15 To: Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

I think that the last parameter "<FQDN>/object-store/", should be only "<FQDN>". Can you test it?

You are using EC2 credentials to authenticate in RGW. Did you enable the Keystone integration in RGW? Also, as far as I know, this admin endpoint needs a RGW admin. I am not sure if the Keystone and RGW integration would enable/make it possible for someone to authenticate as an admin in RGW. Can you check it? To see if you can call that endpoint with these credentials.

On Wed, Sep 28, 2022 at 6:01 AM Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> wrote: Pollster YML configuration :

--- - name: "dynamic.radosgw.usage" sample_type: "gauge" unit: "B" value_attribute: "total.size" url_path: http://<FQDN>/object-store/admin/usagehttp://%3cFQDN%3e/object-store/admin/usage module: "awsauth" authentication_object: "S3Auth" authentication_parameters: <ACCESS_KEY>,<SECRET_KEY>,<FQDN>/object-store/ user_id_attribute: "user" project_id_attribute: "user" resource_id_attribute: "user" response_entries_key: "summary"

ACCESS_KEY and SECRET_KEY have been created with “openstack ec2 credentials create”.

Ceilometer central is deployed with OSA and it uses awsauth.py module.

From: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Sent: mercredi, 28 septembre 2022 02:01 To: Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

Can you show your YML configuration? Also, did you install the AWS authentication module in the container/host where Ceilometer central is running?

On Mon, Sep 26, 2022 at 12:58 PM Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> wrote: Hello Rafael,

Thanks for the information about ceilometer patches but for now I’m testing with the credentials in the dynamic pollster config file. I will use barbican when I push all this to production.

The keystone authentication performed by the rados gw with the credentials provided by ceilometer still does not work. I wonder if this could be a S3 signature version issue on ceilometer side, that is on S3 client side. This kind of issue exists with the s3 client “s3cmd” and you have to add “—signature-v2” so that “s3cmd” works well.

What do you think ? Do you know which version of S3 signature ceilometer uses while authenticating ?

From: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Sent: mercredi, 7 septembre 2022 19:23 To: Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

Jean, there are two problems with the Ceilometer. I just opened the patches to resolve it: - https://review.opendev.org/c/openstack/ceilometer/+/856305 - https://review.opendev.org/c/openstack/ceilometer/+/856304

Without these patches, you might have problems to use Ceilometer with Non-OpenStack dynamic pollsters and barbican credentials.

On Wed, Aug 31, 2022 at 3:55 PM Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> wrote: It is the RGW user that you have. This user must have the role that is needed to access the usage feature in RGW. If I am not mistaken, it required an admin user.

On Wed, Aug 31, 2022 at 1:54 PM Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> wrote: Thanks to your help, I am close to the goal. Dynamic pollster is loaded and triggered.

But I get a “Status[403] and reason [Forbidden]” in ceilometer logs while requesting admin/usage.

I’m not sure to understand well the auth mechanism. Are we talking about keystone credentials, ec2 credentials, Rados GW user ?...

For now, in testing phase, I use “authentication_parameters”, not barbican.

-JF

From: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Sent: mardi, 30 août 2022 14:17 To: Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

Yes, you will need to enable the metric/pollster to be processed. That is done via "polling.yml" file. Also, do not forget that you will need to configure Ceilometer to push this new metric. If you use Gnocchi as the backend, you will need to change/update the gnocchi resource YML file. That file maps resources and metrics in the Gnocchi backend. The configuration resides in Ceilometer. You can create/define new resource types and map them to specific metrics. It depends on how you structure your solution.

P.S. You do not need to use "authentication_parameters". You can use the barbican integration to avoid setting your credentials in a file.

On Tue, Aug 30, 2022 at 9:11 AM Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> wrote: Hello,

I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer logs, that it’s actually loaded. But it looks like it was not triggered, I see no trace of ceilometer connection in Rados GW logs.

My definition:

- name: "dynamic.radosgw.usage" sample_type: "gauge" unit: "B" value_attribute: "total.size" url_path: http://<FQDN>/object-store/swift/v1/admin/usagehttp://%3cFQDN%3e/object-store/swift/v1/admin/usage module: "awsauth" authentication_object: "S3Auth" authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN> user_id_attribute: "admin" project_id_attribute: "admin" resource_id_attribute: "admin" response_entries_key: "summary"

Do I have to set an option in ceilometer.conf, or elsewhere, to get my Rados GW dynamic pollster triggered ?

-JF

From: Taltavull Jean-François Sent: lundi, 29 août 2022 18:41 To: 'Rafael Weingärtner' <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

Thanks a lot for your quick answer, Rafael ! I will explore this approach.

Jean-Francois

From: Rafael Weingärtner <rafaelweingartner@gmail.commailto:rafaelweingartner@gmail.com> Sent: lundi, 29 août 2022 17:54 To: Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> Cc: openstack-discuss <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

EXTERNAL MESSAGE - This email comes from outside ELCA companies.

You could use a different approach. You can use Dynamic pollster [1], and create your own mechanism to collect data, without needing to change Ceilometer code. Basically all hard-coded pollsters can be converted to a dynamic pollster that is defined in YML.

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollste...

On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François <jean-francois.taltavull@elca.chmailto:jean-francois.taltavull@elca.ch> wrote: Hi All,

In our OpenStack deployment, API endpoints are defined by using URLs instead of port numbers and HAProxy forwards requests to the right bakend after having ACLed the URL.

In the case of our object-store service, based on RadosGW, the internal API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>https://%3cFQDN%3e/object-store/swift/v1/AUTH_%3ctenant_id%3e"

When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API with the object-store internal endpoint, the URL becomes https://<FQDN>/adminhttps://%3cFQDN%3e/admin, as shown by HAProxy logs. This URL does not match any API endpoint from HAProxy point of view. The line of code that rewrites the URL is this one: https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilomete...

What would you think of adding a mechanism based on new Ceilometer configuration option(s) to control the URL rewriting ?

Our deployment characteristics: - OpenStack release: Wallaby - Ceph and RadosGW version: 15.2.16 - deployment tool: OSA 23.2.1 and ceph-ansible

Best regards, Jean-Francois

-- Rafael Weingärtner

-- Rafael Weingärtner

-- Rafael Weingärtner

-- Rafael Weingärtner

-- Rafael Weingärtner

-- Rafael Weingärtner

-- Rafael Weingärtner

-- Rafael Weingärtner

-- Rafael Weingärtner

-- Rafael Weingärtner

Glad to hear it! If you need something else, just let me know.

On Mon, Oct 10, 2022 at 12:35 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hi Rafaël,

I finally found the cause and it was on my side. I fixed the setup (ceilometer, radosgw pollsters and haproxy) and keystone auth now works fine.

I use the Rados GW ‘rgw_admin_entry’ variable, in particular.

Thanks a lot for helping and for the time you spent on this issue.

JF

*From:* Taltavull Jean-François *Sent:* mardi, 4 octobre 2022 14:33 *To:* 'Rafael Weingärtner' rafaelweingartner@gmail.com *Cc:* 'openstack-discuss' openstack-discuss@lists.openstack.org *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

Hello Raphaël,

I restored the RGW keystone authentication and did some more tests. The problem is that the S3 request signature provided by ceilometer and the one computed by keystone mismatch.

OpenStack release is Wallaby.

keystone/api/s3tokens.py:

class S3Resource(EC2_S3_Resource.ResourceBase):

    @staticmethod

    def _check_signature(creds_ref, credentials):

        string_to_sign =
base64.urlsafe_b64decode(str(credentials['token']))



        if string_to_sign[0:4] != b'AWS4':

            signature = _calculate_signature_v1(string_to_sign,

                                                creds_ref['secret'])

        else:

            signature = _calculate_signature_v4(string_to_sign,

                                                creds_ref['secret'])

        if not utils.auth_str_equal(credentials['signature'], signature):

            raise exception.Unauthorized(
<<<------------------------------------------we fall
there


                message=_('Credential signature mismatch'))

*From:* Taltavull Jean-François *Sent:* vendredi, 30 septembre 2022 14:48 *To:* 'Rafael Weingärtner' rafaelweingartner@gmail.com *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

$ sudo /usr/bin/radosgw --version

ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus
(stable)

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* vendredi, 30 septembre 2022 12:37 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

No, I just showed you the code, so you can see how the authentication is being executed, and where/how the parameters are set in the headers. It is a bit odd, I have used this so many times, and it always works. What is your RGW instance version?

On Fri, Sep 30, 2022 at 4:09 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Do you mean the issue comes from how the `awsauth` module handles the signature ?

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* jeudi, 29 septembre 2022 17:23 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

This is the signature used by the `awsauth` library:

def get_signature(self, r):
    canonical_string = self.get_canonical_string(
        r.url, r.headers, r.method)
    if py3k:
        key = self.secret_key.encode('utf-8')
        msg = canonical_string.encode('utf-8')
    else:
        key = self.secret_key
        msg = canonical_string
    h = hmac.new(key, msg, digestmod=sha)
    return encodestring(h.digest()).strip()

After that is generated, it is added in the headers:

# Create date header if it is not created yet. if 'date' not in r.headers and 'x-amz-date' not in r.headers: r.headers['date'] = formatdate( timeval=None, localtime=False, usegmt=True) signature = self.get_signature(r) if py3k: signature = signature.decode('utf-8') r.headers['Authorization'] = 'AWS %s:%s' % (self.access_key, signature)

On Thu, Sep 29, 2022 at 9:15 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

$ python test_creds.py

Executing test on: [FQDN/object-store/].

Rados GW admin context [/admin] and path [/usage?stats=True] used.

Rados GW request URL [http://FQDN/object-store/admin/bucket?stats=True].

Rados GW host: FQDN

Traceback (most recent call last):

  File "test_creds.py", line 45, in <module>

    raise RGWAdminAPIFailed(

__main__.RGWAdminAPIFailed: RGW AdminOps API returned 403 Forbidden

So the same as with ceilometer. Auth is done by RGW, not by keystone, and the ceph “admin” user exists and owns the right privileges:

$ sudo radosgw-admin user info --uid
admin
                                                                                   [22/296]{

    "user_id": "admin",

    "display_name": "admin user",

    "email": "",

    "suspended": 0,

    "max_buckets": 1000,

    "subusers": [],

    "keys": [

        {

            "user": "admin",

            "access_key": “admin_access_key",

            "secret_key": "admin_secret_key"

        }

    ],

    "swift_keys": [],

    "caps": [

        {

            "type": "buckets",

            "perm": "*"

        },

        {

            "type": "metadata",

            "perm": "*"

        },


{
                                                                                                                                                                          "type":
"usage",
                                                                                                                                                                                         "perm":
"*"
                                                                                                                                                                                                    },
                                                                                                                                                                                                                   {

            "type": "users",
                                                                                                                                                                                                                                                "perm":
"*"
                                                                                                                                                                                                                                                           }
                                                                                                                                                                                                                                                                      ],

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* jeudi, 29 septembre 2022 12:32 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you test you credentials with the following code?

import json
import requests
import os

import six.moves.urllib.parse as urlparse


class RGWAdminAPIFailed(Exception):
    pass


if __name__ == '__main__':

    rados_gw_base_url = "put your RGW URL here. E.g.
http://server.com:port/something"
    print("Executing test on: [%s]." % rados_gw_base_url)

    rados_gw_admin_context = "/admin"

    rados_gw_path = "/usage?stats=True"

    print("Rados GW admin context [%s] and path [%s] used." %
(rados_gw_admin_context, rados_gw_path))

    rados_gw_request_url = urlparse.urljoin(rados_gw_base_url, '/admin') +
'/bucket?stats=True'
    print("Rados GW request URL [%s]." % rados_gw_request_url)

    rados_gw_access_key_to_use = "put your access key here"
    rados_gw_secret_key_to_use = "put your secret key here"

    rados_gw_host_name = urlparse.urlparse(rados_gw_request_url).netloc
    print("Rados GW host: %s" % rados_gw_host_name)
    module_name = "awsauth"
    class_name = "S3Auth"
    arguments = [rados_gw_access_key_to_use, rados_gw_secret_key_to_use,
rados_gw_host_name]
    module = __import__(module_name)
    class_ = getattr(module, class_name)
    instance = class_(*arguments)

    r = requests.get(
        rados_gw_request_url,
        auth=instance, timeout=30)
        #auth=awsauth.S3Auth(*arguments))


    if r.status_code != 200:
        raise RGWAdminAPIFailed(
            ('RGW AdminOps API returned %(status)s %(reason)s') %
            {'status': r.status_code, 'reason': r.reason})

    response_body = r.text
    parsed_json = json.loads(response_body)

    print("Response cookies: [%s]." % r.cookies)

    radosGw_output_file = "/home/<user_here>/Downloads/radosGw-usage.json"

    if os.path.exists(radosGw_output_file):
        os.remove(radosGw_output_file)

    with open(radosGw_output_file, "w") as file1:
        file1.writelines(json.dumps(parsed_json, indent=4, sort_keys=True))
        file1.flush()

    exit(0)

On Thu, Sep 29, 2022 at 4:09 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

python

Python 3.8.10 (default, Sep 28 2021, 16:10:42)

[GCC 9.3.0] on linux

Type "help", "copyright", "credits" or "license" for more information.

...

...

...

import awsauth

...

...

...

awsauth

<module 'awsauth' from '/openstack/venvs/ceilometer-23.2.0/lib/python3.8/site-packages/awsauth.py'>

...

...

...

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 18:40 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you also execute the following:

python



import awsauth



awsauth

That will output a path, and then you can `cat <path>`, example: `cat /var/lib/kolla/venv/lib/python3.8/site-packages/awsauth.py`

On Wed, Sep 28, 2022 at 1:21 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

I removed trailing ‘/object-store/’ from the last value of authentication_parameters

I also:

• disabled s3 keystone auth in RGW

• created a RGW “admin” user with the right privileges to allow admin API

calls

• put RGW in debug mode

And here is what I get in RGW logs:

get_usage string_to_sign=GET Wed, 28 Sep 2022 16:15:45 GMT /admin/usage

get_usage server signature=BlaBlaBlaBla

get_usage client signature=BloBloBlo

get_usage compare=-75

get_usage rgw::auth::s3::LocalEngine denied with reason=-2027

get_usage rgw::auth::s3::AWSAuthStrategy denied with reason=-2027

get_usage rgw::auth::StrategyRegistry::s3_main_strategy_t: trying rgw::auth::s3::AWSAuthStrategy

get_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 13:15 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

I think that the last parameter "<FQDN>/object-store/", should be only " <FQDN>". Can you test it?

You are using EC2 credentials to authenticate in RGW. Did you enable the Keystone integration in RGW?

Also, as far as I know, this admin endpoint needs a RGW admin. I am not sure if the Keystone and RGW integration would enable/make it possible for someone to authenticate as an admin in RGW. Can you check it? To see if you can call that endpoint with these credentials.

On Wed, Sep 28, 2022 at 6:01 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Pollster YML configuration :

---

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: <ACCESS_KEY>,<SECRET_KEY>,<FQDN>/object-store/

  user_id_attribute: "user"

  project_id_attribute: "user"

  resource_id_attribute: "user"

  response_entries_key: "summary"

ACCESS_KEY and SECRET_KEY have been created with “openstack ec2 credentials create”.

Ceilometer central is deployed with OSA and it uses awsauth.py module.

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 28 septembre 2022 02:01 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Can you show your YML configuration? Also, did you install the AWS authentication module in the container/host where Ceilometer central is running?

On Mon, Sep 26, 2022 at 12:58 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello Rafael,

Thanks for the information about ceilometer patches but for now I’m testing with the credentials in the dynamic pollster config file. I will use barbican when I push all this to production.

The keystone authentication performed by the rados gw with the credentials provided by ceilometer still does not work. I wonder if this could be a S3 signature version issue on ceilometer side, that is on S3 client side. This kind of issue exists with the s3 client “s3cmd” and you have to add “—signature-v2” so that “s3cmd” works well.

What do you think ? Do you know which version of S3 signature ceilometer uses while authenticating ?

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mercredi, 7 septembre 2022 19:23 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Jean, there are two problems with the Ceilometer. I just opened the patches to resolve it:

• https://review.opendev.org/c/openstack/ceilometer/+/856305

• https://review.opendev.org/c/openstack/ceilometer/+/856304

Without these patches, you might have problems to use Ceilometer with Non-OpenStack dynamic pollsters and barbican credentials.

On Wed, Aug 31, 2022 at 3:55 PM Rafael Weingärtner < rafaelweingartner@gmail.com> wrote:

It is the RGW user that you have. This user must have the role that is needed to access the usage feature in RGW. If I am not mistaken, it required an admin user.

On Wed, Aug 31, 2022 at 1:54 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Thanks to your help, I am close to the goal. Dynamic pollster is loaded and triggered.

But I get a “Status[403] and reason [Forbidden]” in ceilometer logs while requesting admin/usage.

I’m not sure to understand well the auth mechanism. Are we talking about keystone credentials, ec2 credentials, Rados GW user ?...

For now, in testing phase, I use “authentication_parameters”, not barbican.

-JF

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* mardi, 30 août 2022 14:17 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

Yes, you will need to enable the metric/pollster to be processed. That is done via "polling.yml" file. Also, do not forget that you will need to configure Ceilometer to push this new metric. If you use Gnocchi as the backend, you will need to change/update the gnocchi resource YML file. That file maps resources and metrics in the Gnocchi backend. The configuration resides in Ceilometer. You can create/define new resource types and map them to specific metrics. It depends on how you structure your solution.

P.S. You do not need to use "authentication_parameters". You can use the barbican integration to avoid setting your credentials in a file.

On Tue, Aug 30, 2022 at 9:11 AM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hello,

I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer logs, that it’s actually loaded. But it looks like it was not triggered, I see no trace of ceilometer connection in Rados GW logs.

My definition:

• name: "dynamic.radosgw.usage"

  sample_type: "gauge"

  unit: "B"

  value_attribute: "total.size"

  url_path: http://<FQDN>/object-store/swift/v1/admin/usage

  module: "awsauth"

  authentication_object: "S3Auth"

  authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN>

  user_id_attribute: "admin"

  project_id_attribute: "admin"

  resource_id_attribute: "admin"

  response_entries_key: "summary"

Do I have to set an option in ceilometer.conf, or elsewhere, to get my Rados GW dynamic pollster triggered ?

-JF

*From:* Taltavull Jean-François *Sent:* lundi, 29 août 2022 18:41 *To:* 'Rafael Weingärtner' rafaelweingartner@gmail.com *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

Thanks a lot for your quick answer, Rafael !

I will explore this approach.

Jean-Francois

*From:* Rafael Weingärtner rafaelweingartner@gmail.com *Sent:* lundi, 29 août 2022 17:54 *To:* Taltavull Jean-François jean-francois.taltavull@elca.ch *Cc:* openstack-discuss openstack-discuss@lists.openstack.org *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number

*EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.

You could use a different approach. You can use Dynamic pollster [1], and create your own mechanism to collect data, without needing to change Ceilometer code. Basically all hard-coded pollsters can be converted to a dynamic pollster that is defined in YML.

[1] https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollste...

On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François < jean-francois.taltavull@elca.ch> wrote:

Hi All,

In our OpenStack deployment, API endpoints are defined by using URLs instead of port numbers and HAProxy forwards requests to the right bakend after having ACLed the URL.

In the case of our object-store service, based on RadosGW, the internal API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>"

When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API with the object-store internal endpoint, the URL becomes https://<FQDN>/admin, as shown by HAProxy logs. This URL does not match any API endpoint from HAProxy point of view. The line of code that rewrites the URL is this one: https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilomete...

What would you think of adding a mechanism based on new Ceilometer configuration option(s) to control the URL rewriting ?

Our deployment characteristics:

• OpenStack release: Wallaby
• Ceph and RadosGW version: 15.2.16
• deployment tool: OSA 23.2.1 and ceph-ansible

Best regards, Jean-Francois

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner

--

Rafael Weingärtner