[Telemetry] Error when sending to prometheus pushgateway

Samuel Mutel samuel.mutel at gmail.com
Fri Aug 14 15:18:31 UTC 2020


Hello,

I didn't find the issue. Somebody could help me ?

Thanks.

Le mer. 8 juil. 2020 à 17:55, Samuel Mutel <samuel.mutel at gmail.com> a
écrit :

> Hello,
>
> Thanks for your help. I tried to test the pushgateway manually and it
> seems to work fine. The pushgateway wrote some things on the stdout.
> But when I start the ceilometer, nothing happens. I tried to change the IP
> to use 127.0.0.1 but nothing.
>
> Here is my ceilometer.conf:
>
>> [DEFAULT]
>> auth_strategy = keystone
>> debug = False
>> event_dispatchers = gnocchi
>> meter_dispatchers = gnocchi
>> transport_url = rabbit://openstack:xxxxxx@xx.xx.x.xx
>> ,openstack:xxxxxxx at xx.xx.x.xx,openstack:xxxxxxxxx at xx.xx.x.xx/
>>
>> [cache]
>> backend = dogpile.cache.memcached
>> enabled = True
>> memcache_servers = xx.xx.x.xx:11211,xx.xx.x.xx:11211,xx.xx.x.xx:11211
>>
>> [keystone_authtoken]
>> auth_type = password
>> auth_uri = https://xxxxxxxxxxxx:5000/v3
>> auth_url = https://xxxxxxxxxxxx:5000
>> memcached_servers = xx.xx.x.xx:11211,xx.xx.x.xx:11211,xx.xx.x.xx:11211
>> password = xxxxxx
>> project_domain_id = default
>> project_name = service
>> region_name = RegionOne
>> user_domain_id = default
>> username = ceilometer
>> www_authenticate_uri = https://xxxxxxxxxxxx:5000
>>
>> [notification]
>> pipelines = meter
>>
>> [oslo_messaging_notifications]
>> driver = messagingv2
>>
>> [oslo_middleware]
>> enable_proxy_headers_parsing = True
>>
>> [publisher]
>> telemetry_secret = xxxxxxxxx
>>
>> [service_credentials]
>> auth_type = password
>> auth_url =https://xxxxxxxxxxxx:5000
>> password = xxxxxxxxx
>> project_domain_id = default
>> project_name = service
>> region_name = RegionOne
>> user_domain_id = default
>> username = ceilometer
>>
>
> Here is my event_pipeline.yaml:
>
>> sources:
>>   - name: meter_file
>>     events:
>>       - "*"
>>     sinks:
>>       - prometheus
>>
>> sinks:
>>   - name: prometheus
>>     publishers:
>>             - prometheus://127.0.0.1:9091/metrics/job/ceilometer
>>
>
> Here is my pipeline.yaml:
>
>> sources:
>>   - name: meter_file
>>     interval: 30
>>     meters:
>>       - "*"
>>     sinks:
>>       - prometheus
>>
>> sinks:
>>   - name: prometheus
>>     publishers:
>>             - prometheus://127.0.0.1:9091/metrics/job/ceilometer
>>
>
> Here is my polling.yaml:
>
>> ---
>> sources:
>>     - name: some_pollsters
>>       interval: 300
>>       meters:
>>         - cpu
>>         - cpu_l3_cache
>>         - memory.usage
>>         - network.incoming.bytes
>>         - network.incoming.packets
>>         - network.outgoing.bytes
>>         - network.outgoing.packets
>>         - disk.device.read.bytes
>>         - disk.device.read.requests
>>         - disk.device.write.bytes
>>         - disk.device.write.requests
>>         - hardware.cpu.util
>>         - hardware.memory.used
>>         - hardware.memory.total
>>         - hardware.memory.buffer
>>         - hardware.memory.cached
>>         - hardware.memory.swap.avail
>>         - hardware.memory.swap.total
>>         - hardware.system_stats.io.outgoing.blocks
>>         - hardware.system_stats.io.incoming.blocks
>>         - hardware.network.ip.incoming.datagrams
>>         - hardware.network.ip.outgoing.datagrams
>>
>
> Here is my ceilometer-rootwrap:
>
>> # Configuration for ceilometer-rootwrap
>> # This file should be owned by (and only-writeable by) the root user
>>
>> [DEFAULT]
>> # List of directories to load filter definitions from (separated by ',').
>> # These directories MUST all be only writeable by root !
>> filters_path=/etc/ceilometer/rootwrap.d,/usr/share/ceilometer/rootwrap
>>
>> # List of directories to search executables in, in case filters do not
>> # explicitely specify a full path (separated by ',')
>> # If not specified, defaults to system PATH environment variable.
>> # These directories MUST all be only writeable by root !
>> exec_dirs=/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/sbin,/usr/local/bin
>>
>> # Enable logging to syslog
>> # Default value is False
>> use_syslog=False
>>
>> # Which syslog facility to use.
>> # Valid values include auth, authpriv, syslog, user0, user1...
>> # Default value is 'syslog'
>> syslog_log_facility=syslog
>>
>> # Which messages to log.
>> # INFO means log all usage
>> # ERROR means only log unsuccessful attempts
>> syslog_log_level=ERROR
>>
>
> What configuration is wrong ?
>
> Le ven. 3 juil. 2020 à 13:53, Matthias Runge <mrunge at matthias-runge.de> a
> écrit :
>
>> Okay, that doesn't really help with debugging though.
>>
>> Method not allowed is returned eg. when the endpoint expected an http
>> push where your browser did an http get (that's correct).
>>
>> What I'd do next is to configure ceilometer to send to a different http
>> endpoint (like a webserver on your workstation, just for debugging
>> purposes).
>>
>> Verify that the push gateway works as expected,
>> https://github.com/prometheus/pushgateway
>> has some curl commands mentioned for debugging purposes.
>>
>>
>> Matthias
>>
>> On 03/07/2020 13:07, Samuel Mutel wrote:
>> > If I go to http://10.60.4.11:9091/metrics/job/ceilometer with the web
>> > browser I receive: Method Not Allowed but i think it's normal.
>> > http://10.60.4.11:9091/metrics is working with metrics.
>> >
>> > The pushgateway and the ceilometer is working on the same host for my
>> > test so no network/firewall issue.
>> >
>> > Logs of the pushgateway is only these ones:
>> > level=info ts=2020-07-03T11:04:35.907Z caller=main.go:83 msg="starting
>> > pushgateway" version="(version=1.2.0, branch=HEAD,
>> > revision=b7e0167e9574f4f88404dde9653ee1d3c940f2eb)"
>> > level=info ts=2020-07-03T11:04:35.908Z caller=main.go:84
>> > build_context="(go=go1.13.8, user=root at 0e823ccfff84,
>> > date=20200311-18:51:01)"
>> > level=info ts=2020-07-03T11:04:35.911Z caller=main.go:137
>> > listen_address=:9091
>> >
>> > Le ven. 3 juil. 2020 à 12:14, Matthias Runge <mrunge at matthias-runge.de
>> > <mailto:mrunge at matthias-runge.de>> a écrit :
>> >
>> >     On 03/07/2020 11:25, Samuel Mutel wrote:
>> >     > Hello,
>> >     >
>> >     > I have two questions about ceilometer (openstack version rocky).
>> >     >
>> >     >   * First of all, it seems that ceilometer is sending metrics
>> >     every hour
>> >     >     and I don't understand why.
>> >     >   * Next, I am not able to setup ceilometer to send metrics to
>> >     >     prometheus pushgateway.
>> >     >
>> >     > Here is my configuration:
>> >     >
>> >     >     sources:
>> >     >       - name: meter_file
>> >     >         interval: 30
>> >     >         meters:
>> >     >           - "*"
>> >     >         sinks:
>> >     >           - prometheus
>> >     >
>> >     >     sinks:
>> >     >       - name: prometheus
>> >     >         publishers:
>> >     >                 -
>> >     prometheus://10.60.4.11:9091/metrics/job/ceilometer
>> >     <http://10.60.4.11:9091/metrics/job/ceilometer>
>> >     >     <http://10.60.4.11:9091/metrics/job/ceilometer>
>> >     >
>> >     >
>> >     > Here is the error I received:
>> >     >
>> >     >     vcpus{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"} 2
>> >     >     # TYPE memory gauge
>> >     >     memory{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"}
>> 2048
>> >     >     # TYPE disk.ephemeral.size gauge
>> >     >
>> >
>>   disk.ephemeral.size{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"}
>> >     >     0
>> >     >     # TYPE disk.root.size gauge
>> >     >
>> >      disk.root.size{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"}
>> 0
>> >     >     : HTTPError: 400 Client Error: Bad Request for url:
>> >     >     http://10.60.4.11:9091/metrics/job/ceilometer
>> >     >     2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>> >     >     Traceback (most recent call last):
>> >     >     2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>> >       File
>> >     >
>>  "/usr/lib/python2.7/dist-packages/ceilometer/publisher/http.py",
>> >     >     line 178, in _do_post
>> >     >     2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>>
>> >     >     res.raise_for_status()
>> >     >     2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>> >       File
>> >     >     "/usr/lib/python2.7/dist-packages/requests/models.py", line
>> >     935, in
>> >     >     raise_for_status
>> >     >     2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>>
>> >     >     raise HTTPError(http_error_msg, response=self)
>> >     >     2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>> >     >     HTTPError: 400 Client Error: Bad Request for url:
>> >     >     http://10.60.4.11:9091/metrics/job/ceilometer
>> >     >     2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>> >     >
>> >     >
>> >     > Thanks for your help on this topic.
>> >
>> >
>> >     Hi,
>> >
>> >     first obvious question:
>> >
>> >     are you sure that there is something listening under
>> >     http://10.60.4.11:9091/metrics/job/ceilometer ?
>> >
>> >     Would you have some error logs from the other side? It seems that
>> >     ceilometer is trying to dispatch as expected.
>> >
>> >     Matthias
>> >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200814/b78ee6d0/attachment-0001.html>


More information about the openstack-discuss mailing list