[Telemetry] Error when sending to prometheus pushgateway
Samuel Mutel
samuel.mutel at gmail.com
Fri Aug 14 15:18:31 UTC 2020
Hello,
I didn't find the issue. Somebody could help me ?
Thanks.
Le mer. 8 juil. 2020 à 17:55, Samuel Mutel <samuel.mutel at gmail.com> a
écrit :
> Hello,
>
> Thanks for your help. I tried to test the pushgateway manually and it
> seems to work fine. The pushgateway wrote some things on the stdout.
> But when I start the ceilometer, nothing happens. I tried to change the IP
> to use 127.0.0.1 but nothing.
>
> Here is my ceilometer.conf:
>
>> [DEFAULT]
>> auth_strategy = keystone
>> debug = False
>> event_dispatchers = gnocchi
>> meter_dispatchers = gnocchi
>> transport_url = rabbit://openstack:xxxxxx@xx.xx.x.xx
>> ,openstack:xxxxxxx at xx.xx.x.xx,openstack:xxxxxxxxx at xx.xx.x.xx/
>>
>> [cache]
>> backend = dogpile.cache.memcached
>> enabled = True
>> memcache_servers = xx.xx.x.xx:11211,xx.xx.x.xx:11211,xx.xx.x.xx:11211
>>
>> [keystone_authtoken]
>> auth_type = password
>> auth_uri = https://xxxxxxxxxxxx:5000/v3
>> auth_url = https://xxxxxxxxxxxx:5000
>> memcached_servers = xx.xx.x.xx:11211,xx.xx.x.xx:11211,xx.xx.x.xx:11211
>> password = xxxxxx
>> project_domain_id = default
>> project_name = service
>> region_name = RegionOne
>> user_domain_id = default
>> username = ceilometer
>> www_authenticate_uri = https://xxxxxxxxxxxx:5000
>>
>> [notification]
>> pipelines = meter
>>
>> [oslo_messaging_notifications]
>> driver = messagingv2
>>
>> [oslo_middleware]
>> enable_proxy_headers_parsing = True
>>
>> [publisher]
>> telemetry_secret = xxxxxxxxx
>>
>> [service_credentials]
>> auth_type = password
>> auth_url =https://xxxxxxxxxxxx:5000
>> password = xxxxxxxxx
>> project_domain_id = default
>> project_name = service
>> region_name = RegionOne
>> user_domain_id = default
>> username = ceilometer
>>
>
> Here is my event_pipeline.yaml:
>
>> sources:
>> - name: meter_file
>> events:
>> - "*"
>> sinks:
>> - prometheus
>>
>> sinks:
>> - name: prometheus
>> publishers:
>> - prometheus://127.0.0.1:9091/metrics/job/ceilometer
>>
>
> Here is my pipeline.yaml:
>
>> sources:
>> - name: meter_file
>> interval: 30
>> meters:
>> - "*"
>> sinks:
>> - prometheus
>>
>> sinks:
>> - name: prometheus
>> publishers:
>> - prometheus://127.0.0.1:9091/metrics/job/ceilometer
>>
>
> Here is my polling.yaml:
>
>> ---
>> sources:
>> - name: some_pollsters
>> interval: 300
>> meters:
>> - cpu
>> - cpu_l3_cache
>> - memory.usage
>> - network.incoming.bytes
>> - network.incoming.packets
>> - network.outgoing.bytes
>> - network.outgoing.packets
>> - disk.device.read.bytes
>> - disk.device.read.requests
>> - disk.device.write.bytes
>> - disk.device.write.requests
>> - hardware.cpu.util
>> - hardware.memory.used
>> - hardware.memory.total
>> - hardware.memory.buffer
>> - hardware.memory.cached
>> - hardware.memory.swap.avail
>> - hardware.memory.swap.total
>> - hardware.system_stats.io.outgoing.blocks
>> - hardware.system_stats.io.incoming.blocks
>> - hardware.network.ip.incoming.datagrams
>> - hardware.network.ip.outgoing.datagrams
>>
>
> Here is my ceilometer-rootwrap:
>
>> # Configuration for ceilometer-rootwrap
>> # This file should be owned by (and only-writeable by) the root user
>>
>> [DEFAULT]
>> # List of directories to load filter definitions from (separated by ',').
>> # These directories MUST all be only writeable by root !
>> filters_path=/etc/ceilometer/rootwrap.d,/usr/share/ceilometer/rootwrap
>>
>> # List of directories to search executables in, in case filters do not
>> # explicitely specify a full path (separated by ',')
>> # If not specified, defaults to system PATH environment variable.
>> # These directories MUST all be only writeable by root !
>> exec_dirs=/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/sbin,/usr/local/bin
>>
>> # Enable logging to syslog
>> # Default value is False
>> use_syslog=False
>>
>> # Which syslog facility to use.
>> # Valid values include auth, authpriv, syslog, user0, user1...
>> # Default value is 'syslog'
>> syslog_log_facility=syslog
>>
>> # Which messages to log.
>> # INFO means log all usage
>> # ERROR means only log unsuccessful attempts
>> syslog_log_level=ERROR
>>
>
> What configuration is wrong ?
>
> Le ven. 3 juil. 2020 à 13:53, Matthias Runge <mrunge at matthias-runge.de> a
> écrit :
>
>> Okay, that doesn't really help with debugging though.
>>
>> Method not allowed is returned eg. when the endpoint expected an http
>> push where your browser did an http get (that's correct).
>>
>> What I'd do next is to configure ceilometer to send to a different http
>> endpoint (like a webserver on your workstation, just for debugging
>> purposes).
>>
>> Verify that the push gateway works as expected,
>> https://github.com/prometheus/pushgateway
>> has some curl commands mentioned for debugging purposes.
>>
>>
>> Matthias
>>
>> On 03/07/2020 13:07, Samuel Mutel wrote:
>> > If I go to http://10.60.4.11:9091/metrics/job/ceilometer with the web
>> > browser I receive: Method Not Allowed but i think it's normal.
>> > http://10.60.4.11:9091/metrics is working with metrics.
>> >
>> > The pushgateway and the ceilometer is working on the same host for my
>> > test so no network/firewall issue.
>> >
>> > Logs of the pushgateway is only these ones:
>> > level=info ts=2020-07-03T11:04:35.907Z caller=main.go:83 msg="starting
>> > pushgateway" version="(version=1.2.0, branch=HEAD,
>> > revision=b7e0167e9574f4f88404dde9653ee1d3c940f2eb)"
>> > level=info ts=2020-07-03T11:04:35.908Z caller=main.go:84
>> > build_context="(go=go1.13.8, user=root at 0e823ccfff84,
>> > date=20200311-18:51:01)"
>> > level=info ts=2020-07-03T11:04:35.911Z caller=main.go:137
>> > listen_address=:9091
>> >
>> > Le ven. 3 juil. 2020 à 12:14, Matthias Runge <mrunge at matthias-runge.de
>> > <mailto:mrunge at matthias-runge.de>> a écrit :
>> >
>> > On 03/07/2020 11:25, Samuel Mutel wrote:
>> > > Hello,
>> > >
>> > > I have two questions about ceilometer (openstack version rocky).
>> > >
>> > > * First of all, it seems that ceilometer is sending metrics
>> > every hour
>> > > and I don't understand why.
>> > > * Next, I am not able to setup ceilometer to send metrics to
>> > > prometheus pushgateway.
>> > >
>> > > Here is my configuration:
>> > >
>> > > sources:
>> > > - name: meter_file
>> > > interval: 30
>> > > meters:
>> > > - "*"
>> > > sinks:
>> > > - prometheus
>> > >
>> > > sinks:
>> > > - name: prometheus
>> > > publishers:
>> > > -
>> > prometheus://10.60.4.11:9091/metrics/job/ceilometer
>> > <http://10.60.4.11:9091/metrics/job/ceilometer>
>> > > <http://10.60.4.11:9091/metrics/job/ceilometer>
>> > >
>> > >
>> > > Here is the error I received:
>> > >
>> > > vcpus{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"} 2
>> > > # TYPE memory gauge
>> > > memory{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"}
>> 2048
>> > > # TYPE disk.ephemeral.size gauge
>> > >
>> >
>> disk.ephemeral.size{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"}
>> > > 0
>> > > # TYPE disk.root.size gauge
>> > >
>> > disk.root.size{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"}
>> 0
>> > > : HTTPError: 400 Client Error: Bad Request for url:
>> > > http://10.60.4.11:9091/metrics/job/ceilometer
>> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>> > > Traceback (most recent call last):
>> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>> > File
>> > >
>> "/usr/lib/python2.7/dist-packages/ceilometer/publisher/http.py",
>> > > line 178, in _do_post
>> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>>
>> > > res.raise_for_status()
>> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>> > File
>> > > "/usr/lib/python2.7/dist-packages/requests/models.py", line
>> > 935, in
>> > > raise_for_status
>> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>>
>> > > raise HTTPError(http_error_msg, response=self)
>> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>> > > HTTPError: 400 Client Error: Bad Request for url:
>> > > http://10.60.4.11:9091/metrics/job/ceilometer
>> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
>> > >
>> > >
>> > > Thanks for your help on this topic.
>> >
>> >
>> > Hi,
>> >
>> > first obvious question:
>> >
>> > are you sure that there is something listening under
>> > http://10.60.4.11:9091/metrics/job/ceilometer ?
>> >
>> > Would you have some error logs from the other side? It seems that
>> > ceilometer is trying to dispatch as expected.
>> >
>> > Matthias
>> >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200814/b78ee6d0/attachment-0001.html>
More information about the openstack-discuss
mailing list