Re: [Telemetry] Error when sending to prometheus pushgateway
Hello, I didn't find the issue. Somebody could help me ? Thanks. Le mer. 8 juil. 2020 à 17:55, Samuel Mutel <samuel.mutel@gmail.com> a écrit :
Hello,
Thanks for your help. I tried to test the pushgateway manually and it seems to work fine. The pushgateway wrote some things on the stdout. But when I start the ceilometer, nothing happens. I tried to change the IP to use 127.0.0.1 but nothing.
Here is my ceilometer.conf:
[DEFAULT] auth_strategy = keystone debug = False event_dispatchers = gnocchi meter_dispatchers = gnocchi transport_url = rabbit://openstack:xxxxxx@xx.xx.x.xx ,openstack:xxxxxxx@xx.xx.x.xx,openstack:xxxxxxxxx@xx.xx.x.xx/
[cache] backend = dogpile.cache.memcached enabled = True memcache_servers = xx.xx.x.xx:11211,xx.xx.x.xx:11211,xx.xx.x.xx:11211
[keystone_authtoken] auth_type = password auth_uri = https://xxxxxxxxxxxx:5000/v3 auth_url = https://xxxxxxxxxxxx:5000 memcached_servers = xx.xx.x.xx:11211,xx.xx.x.xx:11211,xx.xx.x.xx:11211 password = xxxxxx project_domain_id = default project_name = service region_name = RegionOne user_domain_id = default username = ceilometer www_authenticate_uri = https://xxxxxxxxxxxx:5000
[notification] pipelines = meter
[oslo_messaging_notifications] driver = messagingv2
[oslo_middleware] enable_proxy_headers_parsing = True
[publisher] telemetry_secret = xxxxxxxxx
[service_credentials] auth_type = password auth_url =https://xxxxxxxxxxxx:5000 password = xxxxxxxxx project_domain_id = default project_name = service region_name = RegionOne user_domain_id = default username = ceilometer
Here is my event_pipeline.yaml:
sources: - name: meter_file events: - "*" sinks: - prometheus
sinks: - name: prometheus publishers: - prometheus://127.0.0.1:9091/metrics/job/ceilometer
Here is my pipeline.yaml:
sources: - name: meter_file interval: 30 meters: - "*" sinks: - prometheus
sinks: - name: prometheus publishers: - prometheus://127.0.0.1:9091/metrics/job/ceilometer
Here is my polling.yaml:
--- sources: - name: some_pollsters interval: 300 meters: - cpu - cpu_l3_cache - memory.usage - network.incoming.bytes - network.incoming.packets - network.outgoing.bytes - network.outgoing.packets - disk.device.read.bytes - disk.device.read.requests - disk.device.write.bytes - disk.device.write.requests - hardware.cpu.util - hardware.memory.used - hardware.memory.total - hardware.memory.buffer - hardware.memory.cached - hardware.memory.swap.avail - hardware.memory.swap.total - hardware.system_stats.io.outgoing.blocks - hardware.system_stats.io.incoming.blocks - hardware.network.ip.incoming.datagrams - hardware.network.ip.outgoing.datagrams
Here is my ceilometer-rootwrap:
# Configuration for ceilometer-rootwrap # This file should be owned by (and only-writeable by) the root user
[DEFAULT] # List of directories to load filter definitions from (separated by ','). # These directories MUST all be only writeable by root ! filters_path=/etc/ceilometer/rootwrap.d,/usr/share/ceilometer/rootwrap
# List of directories to search executables in, in case filters do not # explicitely specify a full path (separated by ',') # If not specified, defaults to system PATH environment variable. # These directories MUST all be only writeable by root ! exec_dirs=/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/sbin,/usr/local/bin
# Enable logging to syslog # Default value is False use_syslog=False
# Which syslog facility to use. # Valid values include auth, authpriv, syslog, user0, user1... # Default value is 'syslog' syslog_log_facility=syslog
# Which messages to log. # INFO means log all usage # ERROR means only log unsuccessful attempts syslog_log_level=ERROR
What configuration is wrong ?
Le ven. 3 juil. 2020 à 13:53, Matthias Runge <mrunge@matthias-runge.de> a écrit :
Okay, that doesn't really help with debugging though.
Method not allowed is returned eg. when the endpoint expected an http push where your browser did an http get (that's correct).
What I'd do next is to configure ceilometer to send to a different http endpoint (like a webserver on your workstation, just for debugging purposes).
Verify that the push gateway works as expected, https://github.com/prometheus/pushgateway has some curl commands mentioned for debugging purposes.
Matthias
On 03/07/2020 13:07, Samuel Mutel wrote:
If I go to http://10.60.4.11:9091/metrics/job/ceilometer with the web browser I receive: Method Not Allowed but i think it's normal. http://10.60.4.11:9091/metrics is working with metrics.
The pushgateway and the ceilometer is working on the same host for my test so no network/firewall issue.
Logs of the pushgateway is only these ones: level=info ts=2020-07-03T11:04:35.907Z caller=main.go:83 msg="starting pushgateway" version="(version=1.2.0, branch=HEAD, revision=b7e0167e9574f4f88404dde9653ee1d3c940f2eb)" level=info ts=2020-07-03T11:04:35.908Z caller=main.go:84 build_context="(go=go1.13.8, user=root@0e823ccfff84, date=20200311-18:51:01)" level=info ts=2020-07-03T11:04:35.911Z caller=main.go:137 listen_address=:9091
Le ven. 3 juil. 2020 à 12:14, Matthias Runge <mrunge@matthias-runge.de <mailto:mrunge@matthias-runge.de>> a écrit :
On 03/07/2020 11:25, Samuel Mutel wrote: > Hello, > > I have two questions about ceilometer (openstack version rocky). > > * First of all, it seems that ceilometer is sending metrics every hour > and I don't understand why. > * Next, I am not able to setup ceilometer to send metrics to > prometheus pushgateway. > > Here is my configuration: > > sources: > - name: meter_file > interval: 30 > meters: > - "*" > sinks: > - prometheus > > sinks: > - name: prometheus > publishers: > - prometheus://10.60.4.11:9091/metrics/job/ceilometer <http://10.60.4.11:9091/metrics/job/ceilometer> > <http://10.60.4.11:9091/metrics/job/ceilometer> > > > Here is the error I received: > > vcpus{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"} 2 > # TYPE memory gauge > memory{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"} 2048 > # TYPE disk.ephemeral.size gauge >
disk.ephemeral.size{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"}
> 0 > # TYPE disk.root.size gauge > disk.root.size{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"}
0
> : HTTPError: 400 Client Error: Bad Request for url: > http://10.60.4.11:9091/metrics/job/ceilometer > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http > Traceback (most recent call last): > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http File >
"/usr/lib/python2.7/dist-packages/ceilometer/publisher/http.py",
> line 178, in _do_post > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
> res.raise_for_status() > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http File > "/usr/lib/python2.7/dist-packages/requests/models.py", line 935, in > raise_for_status > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http
> raise HTTPError(http_error_msg, response=self) > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http > HTTPError: 400 Client Error: Bad Request for url: > http://10.60.4.11:9091/metrics/job/ceilometer > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http > > > Thanks for your help on this topic.
Hi,
first obvious question:
are you sure that there is something listening under http://10.60.4.11:9091/metrics/job/ceilometer ?
Would you have some error logs from the other side? It seems that ceilometer is trying to dispatch as expected.
Matthias
participants (1)
-
Samuel Mutel