[ceilometer] radosgw pollster
Florian Engelmann
florian.engelmann at everyware.ch
Thu Feb 28 10:58:29 UTC 2019
Hi Christian,
all good! I was not getting any result with "openstack resource list" as
we got to many resources.
openstack metric resource list --type ceph_account -c id -f value | wc -l
gives results!!!!
Looks like everything is working as expected!
Thank you so much for your help!
All the best,
Florian
Am 2/28/19 um 11:21 AM schrieb Florian Engelmann:
> Hi Christian,
>
> after adding requests-aws to the container the PollsterPermanentError is
> gone and the ceilometer polling log looks clean to me:
>
> 2019-02-28 10:29:46.163 24 INFO ceilometer.polling.manager [-] Polling
> pollster radosgw.containers.objects in the context of
> radosgw_300s_pollsters
> 2019-02-28 10:29:46.167 24 INFO ceilometer.polling.manager [-] Polling
> pollster radosgw.objects in the context of radosgw_300s_pollsters
> 2019-02-28 10:29:46.172 24 INFO ceilometer.polling.manager [-] Polling
> pollster radosgw.objects.size in the context of radosgw_300s_pollsters
> 2019-02-28 10:29:46.177 24 INFO ceilometer.polling.manager [-] Polling
> pollster radosgw.objects.containers in the context of
> radosgw_300s_pollsters
> 2019-02-28 10:29:46.182 24 INFO ceilometer.polling.manager [-] Polling
> pollster radosgw.usage in the context of radosgw_300s_pollsters
>
>
> I still do not get any data in gnocchi:
>
> openstack metric resource list -c type -f value | sort -u
> generic
> image
> instance
> instance_disk
> instance_network_interface
> network
> volume
>
> My RadosGW logs do show the poller requests:
> 2019-02-28 10:29:54.767266 7f67a550e700 20 HTTP_ACCEPT=*/*
> 2019-02-28 10:29:54.767272 7f67a550e700 20 HTTP_ACCEPT_ENCODING=gzip,
> deflate
> 2019-02-28 10:29:54.767274 7f67a550e700 20 HTTP_AUTHORIZATION=AWS
> 0APxxxxxxxxx:vxxxxxxxICq/qQXxxxxxxxxxuuc=
> [...]
> 2019-02-28 10:29:54.767294 7f67a550e700 20 REMOTE_ADDR=10.xx.xxx.xxx
> 2019-02-28 10:29:54.767295 7f67a550e700 20 REQUEST_METHOD=GET
> 2019-02-28 10:29:54.767296 7f67a550e700 20 REQUEST_URI=/admin/usage
> 2019-02-28 10:29:54.767297 7f67a550e700 20 SCRIPT_URI=/admin/usage
> [...]
> in_hosted_domain_s3website=0 s->info.domain=rgw.xxxxxx
> s->info.request_uri=/admin/usage
> 2019-02-28 10:29:54.767479 7f67a550e700 10 handler=16RGWHandler_Usage
> 2019-02-28 10:29:54.767491 7f67a550e700 2 req 215268:0.000183::GET
> /admin/usage::getting op 0
> 2019-02-28 10:29:54.767495 7f67a550e700 10 op=15RGWOp_Usage_Get
> 2019-02-28 10:29:54.767496 7f67a550e700 2 req 215268:0.000196::GET
> /admin/usage:get_usage:verifying requester
> 2019-02-28 10:29:54.767528 7f67a550e700 20
> rgw::auth::StrategyRegistry::s3_main_strategy_t: trying
> rgw::auth::s3::AWSAuthStrategy
> 2019-02-28 10:29:54.767530 7f67a550e700 20
> rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::S3AnonymousEngine
> 2019-02-28 10:29:54.767558 7f67a550e700 20
> rgw::auth::s3::S3AnonymousEngine denied with reason=-1
> 2019-02-28 10:29:54.767561 7f67a550e700 20
> rgw::auth::s3::AWSAuthStrategy: trying
> rgw::auth::s3::AWSv2ExternalAuthStrategy
> 2019-02-28 10:29:54.767573 7f67a550e700 20
> rgw::auth::s3::AWSv2ExternalAuthStrategy: trying
> rgw::auth::keystone::EC2Engine
> 2019-02-28 10:29:54.767602 7f67a550e700 10 get_canon_resource():
> dest=/admin/usage
> 2019-02-28 10:29:54.767605 7f67a550e700 10 string_to_sign:
> 2019-02-28 10:29:54.767630 7f67a550e700 20 sending request to
> https://keystone.xxxxxxxxx/v3/auth/tokens
> 2019-02-28 10:29:54.767678 7f67a550e700 20 ssl verification is set to off
> 2019-02-28 10:29:55.194088 7f67a550e700 20 sending request to
> https://keystone.xxxxxxxx/v3/s3tokens
> 2019-02-28 10:29:55.194115 7f67a550e700 20 ssl verification is set to off
> 2019-02-28 10:29:55.226841 7f67a550e700 20
> rgw::auth::keystone::EC2Engine denied with reason=-2028
> 2019-02-28 10:29:55.226852 7f67a550e700 20
> rgw::auth::s3::AWSv2ExternalAuthStrategy denied with reason=-2028
> 2019-02-28 10:29:55.226855 7f67a550e700 20
> rgw::auth::s3::AWSAuthStrategy: trying rgw::auth::s3::LocalEngine
> 2019-02-28 10:29:55.226878 7f67a550e700 10 get_canon_resource():
> dest=/admin/usage
> 2019-02-28 10:29:55.226881 7f67a550e700 10 string_to_sign:
> 2019-02-28 10:29:55.226943 7f67a550e700 15 string_to_sign=GET
> [...]
> 2019-02-28 10:29:55.226972 7f67a550e700 15 compare=0
> 2019-02-28 10:29:55.227099 7f67a550e700 20 rgw::auth::s3::LocalEngine
> granted access
> 2019-02-28 10:29:55.227103 7f67a550e700 20
> rgw::auth::s3::AWSAuthStrategy granted access
> 2019-02-28 10:29:55.227106 7f67a550e700 2 req 215268:0.459806::GET
> /admin/usage:get_usage:normalizing buckets and tenants
> 2019-02-28 10:29:55.227109 7f67a550e700 2 req 215268:0.459809::GET
> /admin/usage:get_usage:init permissions
> 2019-02-28 10:29:55.227136 7f67a550e700 2 req 215268:0.459826::GET
> /admin/usage:get_usage:recalculating target
> 2019-02-28 10:29:55.227140 7f67a550e700 2 req 215268:0.459841::GET
> /admin/usage:get_usage:reading permissions
> 2019-02-28 10:29:55.227142 7f67a550e700 2 req 215268:0.459842::GET
> /admin/usage:get_usage:init op
> 2019-02-28 10:29:55.227143 7f67a550e700 2 req 215268:0.459844::GET
> /admin/usage:get_usage:verifying op mask
> 2019-02-28 10:29:55.227145 7f67a550e700 20 required_mask= 0 user.op_mask=7
> 2019-02-28 10:29:55.227146 7f67a550e700 2 req 215268:0.459846::GET
> /admin/usage:get_usage:verifying op permissions
> 2019-02-28 10:29:55.227148 7f67a550e700 2 req 215268:0.459848::GET
> /admin/usage:get_usage:verifying op params
> 2019-02-28 10:29:55.227149 7f67a550e700 2 req 215268:0.459850::GET
> /admin/usage:get_usage:pre-executing
> 2019-02-28 10:29:55.227150 7f67a550e700 2 req 215268:0.459851::GET
> /admin/usage:get_usage:executing
> 2019-02-28 10:29:55.232830 7f67a550e700 2 req 215268:0.465530::GET
> /admin/usage:get_usage:completing
> 2019-02-28 10:29:55.232858 7f67a550e700 2 req 215268:0.465559::GET
> /admin/usage:get_usage:op status=0
> 2019-02-28 10:29:55.232863 7f67a550e700 2 req 215268:0.465564::GET
> /admin/usage:get_usage:http status=200
> 2019-02-28 10:29:55.232865 7f67a550e700 1 ====== req done
> req=0x7f67a55080a0 op status=0 http_status=200 ======
> 2019-02-28 10:29:55.232894 7f67a550e700 1 civetweb: 0x55ced7bd2000:
> 10.0.81.59 - - [28/Feb/2019:10:29:54 +0100] "GET
> /admin/usage?uid=2534a3e876ee41f088098fxxxxxxxxx%242534a3e876ee41f088098f53xxxxxxx
> HTTP/1.1" 200 0 - python-requests/2.19.1
>
>
> Using curl to do the same request gives me the correct usage results:
> {"entries":[{"user":"a772e4abxxxxxxxx4559d$a772e4ab888exxxxxxxf4559d","buckets":[{"bucket":"","time":"2019-02-26
> 12:00:00.000000Z","epoch":1551182400,"owner":"a772e4ab88xxxxxxx88e4f039b3430d688f4559d","categories":[{"category":"list_buckets","bytes_sent":36,"bytes_received":0,"ops":3,"successful_ops":0}]},{"bucket":"-","time":"2019-02-17
> 15:00:00.000000Z","epoch":1550415600,"owner":"a772e4ab88xxxxxxxa772e4ab888e4f039b3430d688f4559d","categories":[{"category":"get_obj","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":0},{"category":"list_bucket","bytes_sent":132,"bytes_received":0,"ops":1,"successful_ops":0}]},{"bucket":"info","time":"2019-02-26
> 12:00:00.000000Z","epoch":1551182400,"owner":"a772e4ab888e4f039xxxxxx72e4ab888e4f039b3430d688f4559d","categories":[{"category":"RGWMovedPermanently","bytes_sent":0,"bytes_received":0,"ops":3,"successful_ops":3}]},{"bucket":"test","time":"2019-02-17
> 15:00:00.000000Z","epoch":1550415600,"owner":"a772e4ab8xxxxxxd$a772e4ab888e4f039b3430d688f4559d","categories":[{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"list_bucket","bytes_sent":1368,"bytes_received":0,"ops":18,"successful_ops":18},{"category":"put_obj","bytes_sent":0,"bytes_received":10,"ops":1,"successful_ops":1}]}]}],"summary":[{"user":"a772e4abxxxx4f0xxx8f4559d$a772e4ab88xxxxxx30d688f4559d","categories":[{"category":"RGWMovedPermanently","bytes_sent":0,"bytes_received":0,"ops":3,"successful_ops":3},{"category":"create_bucket","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":1},{"category":"get_obj","bytes_sent":0,"bytes_received":0,"ops":1,"successful_ops":0},{"category":"list_bucket","bytes_sent":1500,"bytes_received":0,"ops":19,"successful_ops":18},{"category":"list_buckets","bytes_sent":36,"bytes_received":0,"ops":3,"successful_ops":0},{"category":"put_obj","bytes_sent":0,"bytes_received":10,"ops":1,"successful_ops":1}],"total":{"bytes_sent":153*
> Connection #0 to host 10.xxx.xxx.xxx left intact
>
>
> My custom archive policy (custom_gnocchi_resources.yaml) looks like:
> [...]
> - resource_type: ceph_account
> metrics:
> radosgw.objects:
> radosgw.objects.size:
> radosgw.objects.containers:
> radosgw.api.request:
> radosgw.containers.objects:
> radosgw.containers.objects.size:
> [...]
>
> and the pipeline:
> [...]
> sources:
> - name: meter_source
> meters:
> - "*"
> sinks:
> - meter_sink
> [...]
> sinks:
> - name: meter_sink
> transformers:
> publishers:
> -
> gnocchi://?resources_definition_file=%2Fetc%2Fceilometer%2Fcustom_gnocchi_resources.yaml
>
> [...]
>
> Anything wrong with my archive policy or the pipeline?
>
> All the best,
> Florian
>
> Am 2/27/19 um 8:38 PM schrieb Engelmann Florian:
>> Hi Christian,
>>
>>
>> looks like a hit:
>>
>>
>> https://github.com/openstack/ceilometer/commit/c9eb2d44df7cafde1294123d66445ebef4cfb76d
>>
>>
>>
>> You made my day!
>>
>>
>> I will test tomorrow and report back!
>>
>>
>>
>> All the best,
>>
>> Florian
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Engelmann Florian <florian.engelmann at everyware.ch>
>> *Sent:* Wednesday, February 27, 2019 8:33 PM
>> *To:* Christian Zunker
>> *Cc:* openstack-discuss at lists.openstack.org
>> *Subject:* Re: [ceilometer] radosgw pollster
>>
>> Hi Christian,
>>
>>
>> thank you for your feedback and help! Permissions are fine as I tried
>> to poll the Endpoint successfully with curl and the user (key +
>> secret) we created (and is configured in ceilometer.conf).
>>
>> I saw the requests-aws is used in OSA and it is indeed missing in the
>> kolla container (we use "source" not binary).
>>
>>
>> https://github.com/openstack/kolla/blob/master/docker/ceilometer/ceilometer-base/Dockerfile.j2
>>
>>
>>
>> I will build a new ceilometer container including requests-aws
>> tomorrow to see if this fixes the problem.
>>
>>
>> All the best,
>>
>> Florian
>>
>>
>> ------------------------------------------------------------------------
>> *From:* Christian Zunker <christian.zunker at codecentric.cloud>
>> *Sent:* Wednesday, February 27, 2019 9:09 AM
>> *To:* Engelmann Florian
>> *Cc:* openstack-discuss at lists.openstack.org
>> *Subject:* Re: [ceilometer] radosgw pollster
>> Hi Florian,
>>
>> have you tried different permissions for your ceilometer user in radosgw?
>> According to the docs you need an admin user:
>> https://docs.openstack.org/ceilometer/latest/admin/telemetry-measurements.html#ceph-object-storage
>>
>> Our user has these caps:
>> usage=read,write;metadata=read,write;users=read,write;buckets=read,write
>>
>> We also had to add the requests-aws pip package to query radosgw from
>> ceilometer:
>> https://docs.openstack.org/openstack-ansible/latest/user/ceph/ceilometer.html
>>
>>
>> Christian
>>
>>
>> Am Di., 26. Feb. 2019 um 13:15 Uhr schrieb Florian Engelmann
>> <florian.engelmann at everyware.ch <mailto:florian.engelmann at everyware.ch>>:
>>
>> Hi Christian,
>>
>> Am 2/26/19 um 11:00 AM schrieb Christian Zunker:
>> > Hi Florian,
>> >
>> > which version of OpenStack are you using?
>> > The radosgw metric names were different in some versions:
>> > https://bugs.launchpad.net/ceilometer/+bug/1726458
>>
>> we do use Rocky and Ceilometer 11.0.1. I am still lost with that
>> error.
>> As far as I am able to understand python it looks like the error is
>> happening in polling.manager line 222:
>>
>>
>> https://github.com/openstack/ceilometer/blob/11.0.1/ceilometer/polling/manager.py#L222
>>
>>
>> But I do not understand why. I tried to enable debug logging but the
>> error does not log any additional information.
>> The poller is not even trying to reach/poll our RadosGWs. Looks like
>> that manger is blocking those polls.
>>
>> All the best,
>> Florian
>>
>>
>> >
>> > Christian
>> >
>> > Am Fr., 22. Feb. 2019 um 17:40 Uhr schrieb Florian Engelmann
>> > <florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>
>> <mailto:florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>>>:
>> >
>> > Hi,
>> >
>> > I failed to poll any usage data from our radosgw. I get
>> >
>> > 2019-02-22 17:23:57.461 24 INFO ceilometer.polling.manager
>> [-] Polling
>> > pollster radosgw.containers.objects in the context of
>> > radosgw_300s_pollsters
>> > 2019-02-22 17:23:57.462 24 ERROR ceilometer.polling.manager
>> [-] Prevent
>> > pollster radosgw.containers.objects from polling [<Project
>> > description=,
>> > domain_id=xx9d9975088a4d93922e1d73c7217b3b, enabled=True,
>> >
>> > [...]
>> >
>> > id=xx90a9b1d4be4d75b4bd08ab8107e4ff, is_domain=False,
>> links={u'self':
>> > u'http://keystone-admin.service.xxxxxxx:35357/v3/projects on
>> source
>> > radosgw_300s_pollsters anymore!: PollsterPermanentError
>> >
>> > Configurations like:
>> > cat polling.yaml
>> > ---
>> > sources:
>> > - name: radosgw_300s_pollsters
>> > interval: 300
>> > meters:
>> > - radosgw.usage
>> > - radosgw.objects
>> > - radosgw.objects.size
>> > - radosgw.objects.containers
>> > - radosgw.containers.objects
>> > - radosgw.containers.objects.size
>> >
>> >
>> > Also tried radosgw.api.requests instead of radowsgw.usage.
>> >
>> > ceilometer.conf
>> > [...]
>> > [service_types]
>> > radosgw = object-store
>> >
>> > [rgw_admin_credentials]
>> > access_key = xxxxx0Z0xxxxxxxxxxxx
>> > secret_key = xxxxxxxxxxxxlRExxcPxxxxxxoNxxxxxxOxxxx
>> >
>> > [rgw_client]
>> > implicit_tenants = true
>> >
>> > Endpoints:
>> > | xxxxxxx | region | swift | object-store | True
>> | admin
>> > |
>> http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s |
>> > | xxxxxxx | region | swift | object-store | True
>> |
>> > internal
>> > |
>> http://rgw.service.internalxxx/swift/v1/AUTH_%(tenant_id)s |
>> > | xxxxxxx | region | swift | object-store | True
>> | public
>> > | https://s3.somedomain.com/swift/v1/AUTH_%(tenant_id)s
>> |
>> >
>> > Ceilometer user:
>> > {
>> > "user_id": "ceilometer",
>> > "display_name": "ceilometer",
>> > "email": "",
>> > "suspended": 0,
>> > "max_buckets": 1000,
>> > "auid": 0,
>> > "subusers": [],
>> > "keys": [
>> > {
>> > "user": "ceilometer",
>> > "access_key": "xxxxxxxxxxxxxxxxxx",
>> > "secret_key": "xxxxxxxxxxxxxxxxxxxxxxxxx"
>> > }
>> > ],
>> > "swift_keys": [],
>> > "caps": [
>> > {
>> > "type": "buckets",
>> > "perm": "read"
>> > },
>> > {
>> > "type": "metadata",
>> > "perm": "read"
>> > },
>> > {
>> > "type": "usage",
>> > "perm": "read"
>> > },
>> > {
>> > "type": "users",
>> > "perm": "read"
>> > }
>> > ],
>> > "op_mask": "read, write, delete",
>> > "default_placement": "",
>> > "placement_tags": [],
>> > "bucket_quota": {
>> > "enabled": false,
>> > "check_on_raw": false,
>> > "max_size": -1,
>> > "max_size_kb": 0,
>> > "max_objects": -1
>> > },
>> > "user_quota": {
>> > "enabled": false,
>> > "check_on_raw": false,
>> > "max_size": -1,
>> > "max_size_kb": 0,
>> > "max_objects": -1
>> > },
>> > "temp_url_keys": [],
>> > "type": "rgw"
>> > }
>> >
>> >
>> > radosgw config:
>> > [client.rgw.xxxxxxxxxxx]
>> > host = somehost
>> > rgw frontends = "civetweb port=7480 num_threads=512"
>> > rgw num rados handles = 8
>> > rgw thread pool size = 512
>> > rgw cache enabled = true
>> > rgw dns name = s3.xxxxxx.xxx
>> > rgw enable usage log = true
>> > rgw usage log tick interval = 30
>> > rgw realm = public
>> > rgw zonegroup = xxx
>> > rgw zone = xxxxx
>> > rgw resolve cname = False
>> > rgw usage log flush threshold = 1024
>> > rgw usage max user shards = 1
>> > rgw usage max shards = 32
>> > rgw_keystone_url = https://keystone.xxxxxxxxxxxxx
>> > rgw_keystone_admin_domain = default
>> > rgw_keystone_admin_project = service
>> > rgw_keystone_admin_user = swift
>> > rgw_keystone_admin_password =
>> > xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
>> > rgw_keystone_accepted_roles = member,_member_,admin
>> > rgw_keystone_accepted_admin_roles = admin
>> > rgw_keystone_api_version = 3
>> > rgw_keystone_verify_ssl = false
>> > rgw_keystone_implicit_tenants = true
>> > rgw_keystone_admin_tenant = default
>> > rgw_keystone_revocation_interval = 0
>> > rgw_keystone_token_cache_size = 0
>> > rgw_s3_auth_use_keystone = true
>> > rgw_max_attr_size = 1024
>> > rgw_max_attrs_num_in_req = 32
>> > rgw_max_attr_name_len = 64
>> > rgw_swift_account_in_url = true
>> > rgw_swift_versioning_enabled = true
>> > rgw_enable_apis = s3,swift,swift_auth,admin
>> > rgw_swift_enforce_content_length = true
>> >
>> >
>> >
>> >
>> > Any idea whats going on?
>> >
>> > All the best,
>> > Florian
>> >
>> >
>> >
>>
>> --
>> EveryWare AG
>> Florian Engelmann
>> Senior UNIX Systems Engineer
>> Zurlindenstrasse 52a
>> CH-8003 Zürich
>>
>> tel: +41 44 466 60 00
>> fax: +41 44 466 60 10
>> mail: mailto:florian.engelmann at everyware.ch
>> <mailto:florian.engelmann at everyware.ch>
>> web: http://www.everyware.ch
>>
>>
>>
>
--
EveryWare AG
Florian Engelmann
Senior UNIX Systems Engineer
Zurlindenstrasse 52a
CH-8003 Zürich
tel: +41 44 466 60 00
fax: +41 44 466 60 10
mail: mailto:florian.engelmann at everyware.ch
web: http://www.everyware.ch
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5230 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20190228/dc9f2836/attachment-0001.bin>
More information about the openstack-discuss
mailing list