[Ceilometer] Pollster cannot get RadosGW metrics when API endpoints are based on URL instead of port number
Rafael Weingärtner
rafaelweingartner at gmail.com
Thu Sep 29 10:31:41 UTC 2022
Can you test you credentials with the following code?
```
import json
import requests
import os
import six.moves.urllib.parse as urlparse
class RGWAdminAPIFailed(Exception):
pass
if __name__ == '__main__':
rados_gw_base_url = "put your RGW URL here. E.g. http://server.com:port
/something"
print("Executing test on: [%s]." % rados_gw_base_url)
rados_gw_admin_context = "/admin"
rados_gw_path = "/usage?stats=True"
print("Rados GW admin context [%s] and path [%s] used." %
(rados_gw_admin_context, rados_gw_path))
rados_gw_request_url = urlparse.urljoin(rados_gw_base_url, '/admin') +
'/bucket?stats=True'
print("Rados GW request URL [%s]." % rados_gw_request_url)
rados_gw_access_key_to_use = "put your access key here"
rados_gw_secret_key_to_use = "put your secret key here"
rados_gw_host_name = urlparse.urlparse(rados_gw_request_url).netloc
print("Rados GW host: %s" % rados_gw_host_name)
module_name = "awsauth"
class_name = "S3Auth"
arguments = [rados_gw_access_key_to_use, rados_gw_secret_key_to_use,
rados_gw_host_name]
module = __import__(module_name)
class_ = getattr(module, class_name)
instance = class_(*arguments)
r = requests.get(
rados_gw_request_url,
auth=instance, timeout=30)
#auth=awsauth.S3Auth(*arguments))
if r.status_code != 200:
raise RGWAdminAPIFailed(
('RGW AdminOps API returned %(status)s %(reason)s') %
{'status': r.status_code, 'reason': r.reason})
response_body = r.text
parsed_json = json.loads(response_body)
print("Response cookies: [%s]." % r.cookies)
radosGw_output_file = "/home/<user_here>/Downloads/radosGw-usage.json"
if os.path.exists(radosGw_output_file):
os.remove(radosGw_output_file)
with open(radosGw_output_file, "w") as file1:
file1.writelines(json.dumps(parsed_json, indent=4, sort_keys=True))
file1.flush()
exit(0)
```
On Thu, Sep 29, 2022 at 4:09 AM Taltavull Jean-François <
jean-francois.taltavull at elca.ch> wrote:
> python
>
> Python 3.8.10 (default, Sep 28 2021, 16:10:42)
>
> [GCC 9.3.0] on linux
>
> Type "help", "copyright", "credits" or "license" for more information.
>
> >>> import awsauth
>
> >>> awsauth
>
> <module 'awsauth' from
> '/openstack/venvs/ceilometer-23.2.0/lib/python3.8/site-packages/awsauth.py'>
>
> >>>
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* mercredi, 28 septembre 2022 18:40
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> Can you also execute the following:
>
> ```
>
> python
>
>
>
> import awsauth
>
>
>
> awsauth
>
> ```
>
> That will output a path, and then you can `cat <path>`, example: `cat
> /var/lib/kolla/venv/lib/python3.8/site-packages/awsauth.py`
>
>
>
> On Wed, Sep 28, 2022 at 1:21 PM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> I removed trailing ‘/object-store/’ from the last value of
> authentication_parameters
>
>
>
> I also:
>
> - disabled s3 keystone auth in RGW
>
> - created a RGW “admin” user with the right privileges to allow admin API
> calls
>
> - put RGW in debug mode
>
>
>
> And here is what I get in RGW logs:
>
>
>
> get_usage
> string_to_sign=GET
> Wed,
> 28 Sep 2022 16:15:45
> GMT
> /admin/usage
>
> get_usage server signature=BlaBlaBlaBla
>
> get_usage client signature=BloBloBlo
>
> get_usage compare=-75
>
> get_usage rgw::auth::s3::LocalEngine denied with reason=-2027
>
> get_usage rgw::auth::s3::AWSAuthStrategy denied with reason=-2027
>
> get_usage rgw::auth::StrategyRegistry::s3_main_strategy_t: trying
> rgw::auth::s3::AWSAuthStrategy
>
> get_usage rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* mercredi, 28 septembre 2022 13:15
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> I think that the last parameter "<FQDN>/object-store/", should be only "
> <FQDN>". Can you test it?
>
>
>
>
>
> You are using EC2 credentials to authenticate in RGW. Did you enable the
> Keystone integration in RGW?
>
> Also, as far as I know, this admin endpoint needs a RGW admin. I am not
> sure if the Keystone and RGW integration would enable/make it possible for
> someone to authenticate as an admin in RGW. Can you check it? To see if you
> can call that endpoint with these credentials.
>
>
>
> On Wed, Sep 28, 2022 at 6:01 AM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> Pollster YML configuration :
>
>
>
> ---
>
> - name: "dynamic.radosgw.usage"
>
> sample_type: "gauge"
>
> unit: "B"
>
> value_attribute: "total.size"
>
> url_path: http://<FQDN>/object-store/admin/usage
>
> module: "awsauth"
>
> authentication_object: "S3Auth"
>
> authentication_parameters: <ACCESS_KEY>,<SECRET_KEY>,<FQDN>/object-store/
>
> user_id_attribute: "user"
>
> project_id_attribute: "user"
>
> resource_id_attribute: "user"
>
> response_entries_key: "summary"
>
>
>
> ACCESS_KEY and SECRET_KEY have been created with “openstack ec2
> credentials create”.
>
>
>
> Ceilometer central is deployed with OSA and it uses awsauth.py module.
>
>
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* mercredi, 28 septembre 2022 02:01
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> Can you show your YML configuration? Also, did you install the AWS
> authentication module in the container/host where Ceilometer central is
> running?
>
>
>
> On Mon, Sep 26, 2022 at 12:58 PM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> Hello Rafael,
>
>
>
> Thanks for the information about ceilometer patches but for now I’m
> testing with the credentials in the dynamic pollster config file. I will
> use barbican when I push all this to production.
>
>
>
> The keystone authentication performed by the rados gw with the credentials
> provided by ceilometer still does not work. I wonder if this could be a S3
> signature version issue on ceilometer side, that is on S3 client side. This
> kind of issue exists with the s3 client “s3cmd” and you have to add
> “—signature-v2” so that “s3cmd” works well.
>
>
>
> What do you think ? Do you know which version of S3 signature ceilometer
> uses while authenticating ?
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* mercredi, 7 septembre 2022 19:23
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> Jean, there are two problems with the Ceilometer. I just opened the
> patches to resolve it:
> - https://review.opendev.org/c/openstack/ceilometer/+/856305
>
> - https://review.opendev.org/c/openstack/ceilometer/+/856304
>
>
>
> Without these patches, you might have problems to use Ceilometer with
> Non-OpenStack dynamic pollsters and barbican credentials.
>
>
>
> On Wed, Aug 31, 2022 at 3:55 PM Rafael Weingärtner <
> rafaelweingartner at gmail.com> wrote:
>
> It is the RGW user that you have. This user must have the role that is
> needed to access the usage feature in RGW. If I am not mistaken, it
> required an admin user.
>
>
>
> On Wed, Aug 31, 2022 at 1:54 PM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> Thanks to your help, I am close to the goal. Dynamic pollster is loaded
> and triggered.
>
>
>
> But I get a “Status[403] and reason [Forbidden]” in ceilometer logs while
> requesting admin/usage.
>
>
>
> I’m not sure to understand well the auth mechanism. Are we talking about
> keystone credentials, ec2 credentials, Rados GW user ?...
>
>
>
> For now, in testing phase, I use “authentication_parameters”, not barbican.
>
>
>
> -JF
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* mardi, 30 août 2022 14:17
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> Yes, you will need to enable the metric/pollster to be processed. That is
> done via "polling.yml" file. Also, do not forget that you will need to
> configure Ceilometer to push this new metric. If you use Gnocchi as the
> backend, you will need to change/update the gnocchi resource YML file. That
> file maps resources and metrics in the Gnocchi backend. The configuration
> resides in Ceilometer. You can create/define new resource types and map
> them to specific metrics. It depends on how you structure your solution.
>
> P.S. You do not need to use "authentication_parameters". You can use the
> barbican integration to avoid setting your credentials in a file.
>
>
>
> On Tue, Aug 30, 2022 at 9:11 AM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> Hello,
>
>
>
> I tried to define a Rados GW dynamic pollster and I can see, in Ceilometer
> logs, that it’s actually loaded. But it looks like it was not triggered, I
> see no trace of ceilometer connection in Rados GW logs.
>
>
>
> My definition:
>
>
>
> - name: "dynamic.radosgw.usage"
>
> sample_type: "gauge"
>
> unit: "B"
>
> value_attribute: "total.size"
>
> url_path: http://<FQDN>/object-store/swift/v1/admin/usage
>
> module: "awsauth"
>
> authentication_object: "S3Auth"
>
> authentication_parameters: xxxxxxxxxxxxx,yyyyyyyyyyyyy,<FQDN>
>
> user_id_attribute: "admin"
>
> project_id_attribute: "admin"
>
> resource_id_attribute: "admin"
>
> response_entries_key: "summary"
>
>
>
> Do I have to set an option in ceilometer.conf, or elsewhere, to get my
> Rados GW dynamic pollster triggered ?
>
>
>
> -JF
>
>
>
> *From:* Taltavull Jean-François
> *Sent:* lundi, 29 août 2022 18:41
> *To:* 'Rafael Weingärtner' <rafaelweingartner at gmail.com>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* RE: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
> Thanks a lot for your quick answer, Rafael !
>
> I will explore this approach.
>
>
>
> Jean-Francois
>
>
>
> *From:* Rafael Weingärtner <rafaelweingartner at gmail.com>
> *Sent:* lundi, 29 août 2022 17:54
> *To:* Taltavull Jean-François <jean-francois.taltavull at elca.ch>
> *Cc:* openstack-discuss <openstack-discuss at lists.openstack.org>
> *Subject:* Re: [Ceilometer] Pollster cannot get RadosGW metrics when API
> endpoints are based on URL instead of port number
>
>
>
>
>
> *EXTERNAL MESSAGE *- This email comes from *outside ELCA companies*.
>
> You could use a different approach. You can use Dynamic pollster [1], and
> create your own mechanism to collect data, without needing to change
> Ceilometer code. Basically all hard-coded pollsters can be converted to a
> dynamic pollster that is defined in YML.
>
>
>
> [1]
> https://docs.openstack.org/ceilometer/latest/admin/telemetry-dynamic-pollster.html#the-dynamic-pollsters-system-configuration-for-non-openstack-apis
>
>
>
>
>
> On Mon, Aug 29, 2022 at 12:51 PM Taltavull Jean-François <
> jean-francois.taltavull at elca.ch> wrote:
>
> Hi All,
>
> In our OpenStack deployment, API endpoints are defined by using URLs
> instead of port numbers and HAProxy forwards requests to the right bakend
> after having ACLed the URL.
>
> In the case of our object-store service, based on RadosGW, the internal
> API endpoint is "https://<FQDN>/object-store/swift/v1/AUTH_<tenant_id>"
>
> When Ceilometer RadosGW pollster tries to connect to the RadosGW admin API
> with the object-store internal endpoint, the URL becomes
> https://<FQDN>/admin, as shown by HAProxy logs. This URL does not match
> any API endpoint from HAProxy point of view. The line of code that rewrites
> the URL is this one:
> https://opendev.org/openstack/ceilometer/src/branch/stable/wallaby/ceilometer/objectstore/rgw.py#L81
>
> What would you think of adding a mechanism based on new Ceilometer
> configuration option(s) to control the URL rewriting ?
>
> Our deployment characteristics:
> - OpenStack release: Wallaby
> - Ceph and RadosGW version: 15.2.16
> - deployment tool: OSA 23.2.1 and ceph-ansible
>
>
> Best regards,
> Jean-Francois
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
>
>
> --
>
> Rafael Weingärtner
>
--
Rafael Weingärtner
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20220929/892da25d/attachment-0001.htm>
More information about the openstack-discuss
mailing list