[Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number
Rafael Weingärtner
rafaelweingartner at gmail.com
Mon Oct 10 15:59:39 UTC 2022
Glad to hear it! If you need something else, just let me know.
On Mon, Oct 10, 2022 at 12:35 PM Taltavull Jean-François <
jean-francois.taltavull at elca.ch> wrote:
> Hi Rafaël,
>
> I finally found the cause and it was on my side. I fixed the setup
> (ceilometer, radosgw pollsters and haproxy) and keystone auth now works
> fine.
>
>
>
> I use the Rados GW ‘rgw_admin_entry’ variable, in particular.
>
>
>
> Thanks a lot for helping and for the time you spent on this issue.
>
>
>
> JF
>
>
>
> *From:* Taltavull Jean-François
> *Sent:* mardi, 4 octobre 2022 14:33
> *To:* 'Rafael Weingärtner' <rafaelweingartner at gmail.com>
> *Cc:* 'openstack-discuss' <openstack-discuss at lists.openstack.org>
> *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
> Hello Raphaël,
>
> I restored the RGW keystone authentication and did some more tests. The
> problem is that the S3 request signature provided by ceilometer and the one
> computed by keystone mismatch.
>
>
>
> OpenStack release is Wallaby.
>
>
>
> keystone/api/s3tokens.py:
>
> ````
>
> class S3Resource(EC2_S3_Resource.ResourceBase):
>
> @staticmethod
>
> def _check_signature(creds_ref, credentials):
>
> string_to_sign =
> base64.urlsafe_b64decode(str(credentials['token']))
>
>
>
> if string_to_sign[0:4] != b'AWS4':
>
> signature = _calculate_signature_v1(string_to_sign,
>
> creds_ref['secret'])
>
> else:
>
> signature = _calculate_signature_v4(string_to_sign,
>
> creds_ref['secret'])
>
> if not utils.auth_str_equal(credentials['signature'], signature):
>
> raise exception.Unauthorized(
> <<<------------------------------------------we fall
> there
>
>
> message=_('Credential signature mismatch'))
> ````
>
>
>
> *From:* Taltavull Jean-François
> *Sent:* vendredi, 30 septembre 2022 14:48
> *To:* 'Rafael Weingärtner' <rafaelweingartner at gmail.com>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
> ```
>
> $ sudo /usr/bin/radosgw --version
>
> ceph version 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus
> (stable)
>
> ```
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* vendredi, 30 septembre 2022 12:37
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> No, I just showed you the code, so you can see how the authentication is
> being executed, and where/how the parameters are set in the headers. It is
> a bit odd, I have used this so many times, and it always works. What is
> your RGW instance version?
>
>
>
> On Fri, Sep 30, 2022 at 4:09 AM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> Do you mean the issue comes from how the `awsauth` module handles the
> signature ?
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* jeudi, 29 septembre 2022 17:23
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> This is the signature used by the `awsauth` library:
> ```
>
> def get_signature(self, r):
> canonical_string = self.get_canonical_string(
> r.url, r.headers, r.method)
> if py3k:
> key = self.secret_key.encode('utf-8')
> msg = canonical_string.encode('utf-8')
> else:
> key = self.secret_key
> msg = canonical_string
> h = hmac.new(key, msg, digestmod=sha)
> return encodestring(h.digest()).strip()
>
>
>
> ```
>
>
>
> After that is generated, it is added in the headers:
>
> # Create date header if it is not created yet.
> if 'date' not in r.headers and 'x-amz-date' not in r.headers:
> r.headers['date'] = formatdate(
> timeval=None,
> localtime=False,
> usegmt=True)
> signature = self.get_signature(r)
> if py3k:
> signature = signature.decode('utf-8')
> r.headers['Authorization'] = 'AWS %s:%s' % (self.access_key, signature)
>
>
>
> On Thu, Sep 29, 2022 at 9:15 AM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> ```
>
> $ python test_creds.py
>
> Executing test on: [FQDN/object-store/].
>
> Rados GW admin context [/admin] and path [/usage?stats=True] used.
>
> Rados GW request URL [http://FQDN/object-store/admin/bucket?stats=True].
>
> Rados GW host: FQDN
>
> Traceback (most recent call last):
>
> File "test_creds.py", line 45, in <module>
>
> raise RGWAdminAPIFailed(
>
> __main__.RGWAdminAPIFailed: RGW AdminOps API returned 403 Forbidden
>
> ```
>
>
>
> So the same as with ceilometer. Auth is done by RGW, not by keystone, and
> the ceph “admin” user exists and owns the right privileges:
>
> ```
>
> $ sudo radosgw-admin user info --uid
> admin
> [22/296]{
>
> "user_id": "admin",
>
> "display_name": "admin user",
>
> "email": "",
>
> "suspended": 0,
>
> "max_buckets": 1000,
>
> "subusers": [],
>
> "keys": [
>
> {
>
> "user": "admin",
>
> "access_key": “admin_access_key",
>
> "secret_key": "admin_secret_key"
>
> }
>
> ],
>
> "swift_keys": [],
>
> "caps": [
>
> {
>
> "type": "buckets",
>
> "perm": "*"
>
> },
>
> {
>
> "type": "metadata",
>
> "perm": "*"
>
> },
>
>
> {
> "type":
> "usage",
> "perm":
> "*"
> },
> {
>
> "type": "users",
> "perm":
> "*"
> }
> ],
>
>
>
>
> ```
>
>
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* jeudi, 29 septembre 2022 12:32
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> Can you test you credentials with the following code?
>
> ```
>
> import json
> import requests
> import os
>
> import six.moves.urllib.parse as urlparse
>
>
> class RGWAdminAPIFailed(Exception):
> pass
>
>
> if __name__ == '__main__':
>
> rados_gw_base_url = "put your RGW URL here. E.g.
> http://server.com:port/something"
> print("Executing test on: [%s]." % rados_gw_base_url)
>
> rados_gw_admin_context = "/admin"
>
> rados_gw_path = "/usage?stats=True"
>
> print("Rados GW admin context [%s] and path [%s] used." %
> (rados_gw_admin_context, rados_gw_path))
>
> rados_gw_request_url = urlparse.urljoin(rados_gw_base_url, '/admin') +
> '/bucket?stats=True'
> print("Rados GW request URL [%s]." % rados_gw_request_url)
>
> rados_gw_access_key_to_use = "put your access key here"
> rados_gw_secret_key_to_use = "put your secret key here"
>
> rados_gw_host_name = urlparse.urlparse(rados_gw_request_url).netloc
> print("Rados GW host: %s" % rados_gw_host_name)
> module_name = "awsauth"
> class_name = "S3Auth"
> arguments = [rados_gw_access_key_to_use, rados_gw_secret_key_to_use,
> rados_gw_host_name]
> module = __import__(module_name)
> class_ = getattr(module, class_name)
> instance = class_(*arguments)
>
> r = requests.get(
> rados_gw_request_url,
> auth=instance, timeout=30)
> #auth=awsauth.S3Auth(*arguments))
>
>
> if r.status_code != 200:
> raise RGWAdminAPIFailed(
> ('RGW AdminOps API returned %(status)s %(reason)s') %
> {'status': r.status_code, 'reason': r.reason})
>
> response_body = r.text
> parsed_json = json.loads(response_body)
>
> print("Response cookies: [%s]." % r.cookies)
>
> radosGw_output_file = "/home/<user_here>/Downloads/radosGw-usage.json"
>
> if os.path.exists(radosGw_output_file):
> os.remove(radosGw_output_file)
>
> with open(radosGw_output_file, "w") as file1:
> file1.writelines(json.dumps(parsed_json, indent=4, sort_keys=True))
> file1.flush()
>
> exit(0)
>
> ```
>
>
>
> On Thu, Sep 29, 2022 at 4:09 AM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> python
>
> Python 3.8.10 (default, Sep 28 2021, 16:10:42)
>
> [GCC 9.3.0] on linux
>
> Type "help", "copyright", "credits" or "license" for more information.
>
> >>> import awsauth
>
> >>> awsauth
>
> <module 'awsauth' from
> '/openstack/venvs/ceilometer-23.2.0/lib/python3.8/site-packages/awsauth.py'>
>
> >>>
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* mercredi, 28 septembre 2022 18:40
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> Can you also execute the following:
>
> ```
>
> python
>
>
>
> import awsauth
>
>
>
> awsauth
>
> ```
>
> That will output a path, and then you can `cat <path>`, example: `cat
> /var/lib/kolla/venv/lib/python3.8/site-packages/awsauth.py`
>
>
>
> On Wed, Sep 28, 2022 at 1:21 PM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> I removed trailing ‘/object-store/’ from the last value of
> authentication_parameters
>
>
>
> I also:
>
> - disabled s3 keystone auth in RGW
>
> - created a RGW “admin” user with the right privileges to allow admin API
> calls
>
> - put RGW in debug mode
>
>
>
> And here is what I get in RGW logs:
>
>
>
> get_usage
> string_to_sign=GET
> Wed,
> 28 Sep 2022 16:15:45
> GMT
> /admin/usage
>
> get_usage server signature=BlaBlaBlaBla
>
> get_usage client signature=BloBloBlo
>
> get_usage compare=-75
>
> get_usage rgw::auth::s3::LocalEngine denied with reason=-2027
>
> get_usage rgw::auth::s3::AWSAuthStrategy denied with reason=-2027
>
> get_usage rgw::auth::StrategyRegistry::s3_main_strategy_t: trying
> rgw::auth::s3::AWSAuthStrategy
>
> get_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* mercredi, 28 septembre 2022 13:15
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> I think that the last parameter "<FQDN>/object-store/", should be only "
> <FQDN>". Can you test it?
>
>
>
>
>
> You are using EC2 credentials to authenticate in RGW. Did you enable the
> Keystone integration in RGW?
>
> Also, as far as I know, this admin endpoint needs a RGW admin. I am not
> sure if the Keystone and RGW integration would enable/make it possible for
> someone to authenticate as an admin in RGW. Can you check it? To see if you
> can call that endpoint with these credentials.
>
>
>
> On Wed, Sep 28, 2022 at 6:01 AM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> Pollster YML configuration :
>
>
>
> ---
>
> - name: "dynamic.radosgw.usage"
>
> sample_type: "gauge"
>
> unit: "B"
>
> value_attribute: "total.size"
>
> url_path: http://<FQDN>/object-store/admin/usage
>
> module: "awsauth"
>
> authentication_object: "S3Auth"
>
> authentication_parameters: <ACCESS_KEY>,<SECRET_KEY>,<FQDN>/object-store/
>
> user_id_attribute: "user"
>
> project_id_attribute: "user"
>
> resource_id_attribute: "user"
>
> response_entries_key: "summary"
>
>
>
> ACCESS_KEY and SECRET_KEY have been created with “openstack ec2
> credentials create”.
>
>
>
> Ceilometer central is deployed with OSA and it uses awsauth.py module.
>
>
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* mercredi, 28 septembre 2022 02:01
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> Can you show your YML configuration? Also, did you install the AWS
> authentication module in the container/host where Ceilometer central is
> running?
>
>
>
> On Mon, Sep 26, 2022 at 12:58 PM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> Hello Rafael,
>
>
>
> Thanks for the information about ceilometer patches but for now I’m
> testing with the credentials in the dynamic pollster config file. I will
> use barbican when I push all this to production.
>
>
>
> The keystone authentication performed by the rados gw with the credentials
> provided by ceilometer still does not work. I wonder if this could be a S3
> signature version issue on ceilometer side, that is on S3 client side. This
> kind of issue exists with the s3 client “s3cmd” and you have to add
> “—signature-v2” so that “s3cmd” works well.
>
>
>
> What do you think ? Do you know which version of S3 signature ceilometer
> uses while authenticating ?
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* mercredi, 7 septembre 2022 19:23
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> Jean, there are two problems with the Ceilometer. I just opened the
> patches to resolve it:
> - https://review.opendev.org/c/openstack/ceilometer/+/856305
>
> - https://review.opendev.org/c/openstack/ceilometer/+/856304
>
>
>
> Without these patches, you might have problems to use Ceilometer with
> Non-OpenStack dynamic pollsters and barbican credentials.
>
>
>
> On Wed, Aug 31, 2022 at 3:55 PM Rafael Weingärtner <
> rafaelweingartner at gmail.com> wrote:
>
> It is the RGW user that you have. This user must have the role that is
> needed to access the usage feature in RGW. If I am not mistaken, it
> required an admin user.
>
>
>
> On Wed, Aug 31, 2022 at 1:54 PM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> Thanks to your help, I am close to the goal. Dynamic pollster is loaded
> and triggered.
>
>
>
> But I get a “Status[403] and reason [Forbidden]” in ceilometer logs while
> requesting admin/usage.
>
>
>
> I’m not sure to understand well the auth mechanism. Are we talking about
> keystone credentials, ec2 credentials, Rados GW user ?...
>
>
>
> For now, in testing phase, I use “authentication_parameters”, not barbican.
>
>
>
> -JF
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* mardi, 30 août 2022 14:17
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> Yes, you will need to enable the metric/pollster to be processed. That is
> done via "polling.yml" file. Also, do not forget that you will need to
> configure Ceilometer to push this new metric. If you use Gnocchi as the
> backend, you will need to change/update the gnocchi resource YML file. That
> file maps resources and metrics in the Gnocchi backend. The configuration
> resides in Ceilometer. You can create/define new resource types and map
> them to specific metrics. It depends on how you structure your solution.
>
> P.S. You do not need to use "authentication_parameters". You can use the
> barbican integration to avoid setting your credentials in a file.
>
>
>
> On Tue, Aug 30, 2022 at 9:11 AM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> Hello,
>
>
>
> I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer
> logs, that it’s actually loaded. But it looks like it was not triggered, I
> see no trace of ceilometer connection in Rados GW logs.
>
>
>
> My definition:
>
>
>
> - name: "dynamic.radosgw.usage"
>
> sample_type: "gauge"
>
> unit: "B"
>
> value_attribute: "total.size"
>
> url_path: http://<FQDN>/object-store/swift/v1/admin/usage
>
> module: "awsauth"
>
> authentication_object: "S3Auth"
>
> authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN>
>
> user_id_attribute: "admin"
>
> project_id_attribute: "admin"
>
> resource_id_attribute: "admin"
>
> response_entries_key: "summary"
>
>
>
> Do I have to set an option in ceilometer.conf, or elsewhere, to get my
> Rados GW dynamic pollster triggered ?
>
>
>
> -JF
>
>
>
> *From:* Taltavull Jean-François
> *Sent:* lundi, 29 août 2022 18:41
> *To:* 'Rafael Weingärtner' <rafaelweingartner at gmail.com>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
> Thanks a lot for your quick answer, Rafael !
>
> I will explore this approach.
>
>
>
> Jean-Francois
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* lundi, 29 août 2022 17:54
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> You could use a different approach. You can use Dynamic pollster [1], and
> create your own mechanism to collect data, without needing to change
> Ceilometer code. Basically all hard-coded pollsters can be converted to a
> dynamic pollster that is defined in YML.
>
>
>
> [1]
> https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollster.html#the-dynamic-pollsters-system-configuration-for-non-openstack-apis
>
>
>
>
>
> On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> Hi All,
>
> In our OpenStack deployment, API endpoints are defined by using URLs
> instead of port numbers and HAProxy forwards requests to the right bakend
> after having ACLed the URL.
>
> In the case of our object-store service, based on RadosGW, the internal
> API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>"
>
> When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API
> with the object-store internal endpoint, the URL becomes
> https://<FQDN>/admin, as shown by HAProxy logs. This URL does not match
> any API endpoint from HAProxy point of view. The line of code that rewrites
> the URL is this one:
> https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilometer/objectstore/rgw.py#L81
>
> What would you think of adding a mechanism based on new Ceilometer
> configuration option(s) to control the URL rewriting ?
>
> Our deployment characteristics:
> - OpenStack release: Wallaby
> - Ceph and RadosGW version: 15.2.16
> - deployment tool: OSA 23.2.1 and ceph-ansible
>
>
> Best regards,
> Jean-Francois
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
--
Rafael Weingärtner
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20221010/bf292433/attachment-0001.htm>
More information about the openstack-discuss
mailing list