August 2020 - openstack-discuss - lists.openstack.org

Re: [Telemetry] Error when sending to prometheus pushgateway
by Samuel Mutel 14 Aug '20

14 Aug '20

Hello, I didn't find the issue. Somebody could help me ? Thanks. Le mer. 8 juil. 2020 à 17:55, Samuel Mutel <samuel.mutel(a)gmail.com> a écrit : > Hello, > > Thanks for your help. I tried to test the pushgateway manually and it > seems to work fine. The pushgateway wrote some things on the stdout. > But when I start the ceilometer, nothing happens. I tried to change the IP > to use 127.0.0.1 but nothing. > > Here is my ceilometer.conf: > >> [DEFAULT] >> auth_strategy = keystone >> debug = False >> event_dispatchers = gnocchi >> meter_dispatchers = gnocchi >> transport_url = rabbit://openstack:xxxxxx@xx.xx.x.xx >> ,openstack:xxxxxxx@xx.xx.x.xx,openstack:xxxxxxxxx@xx.xx.x.xx/ >> >> [cache] >> backend = dogpile.cache.memcached >> enabled = True >> memcache_servers = xx.xx.x.xx:11211,xx.xx.x.xx:11211,xx.xx.x.xx:11211 >> >> [keystone_authtoken] >> auth_type = password >> auth_uri = https://xxxxxxxxxxxx:5000/v3 >> auth_url = https://xxxxxxxxxxxx:5000 >> memcached_servers = xx.xx.x.xx:11211,xx.xx.x.xx:11211,xx.xx.x.xx:11211 >> password = xxxxxx >> project_domain_id = default >> project_name = service >> region_name = RegionOne >> user_domain_id = default >> username = ceilometer >> www_authenticate_uri = https://xxxxxxxxxxxx:5000 >> >> [notification] >> pipelines = meter >> >> [oslo_messaging_notifications] >> driver = messagingv2 >> >> [oslo_middleware] >> enable_proxy_headers_parsing = True >> >> [publisher] >> telemetry_secret = xxxxxxxxx >> >> [service_credentials] >> auth_type = password >> auth_url =https://xxxxxxxxxxxx:5000 >> password = xxxxxxxxx >> project_domain_id = default >> project_name = service >> region_name = RegionOne >> user_domain_id = default >> username = ceilometer >> > > Here is my event_pipeline.yaml: > >> sources: >> - name: meter_file >> events: >> - "*" >> sinks: >> - prometheus >> >> sinks: >> - name: prometheus >> publishers: >> - prometheus://127.0.0.1:9091/metrics/job/ceilometer >> > > Here is my pipeline.yaml: > >> sources: >> - name: meter_file >> interval: 30 >> meters: >> - "*" >> sinks: >> - prometheus >> >> sinks: >> - name: prometheus >> publishers: >> - prometheus://127.0.0.1:9091/metrics/job/ceilometer >> > > Here is my polling.yaml: > >> --- >> sources: >> - name: some_pollsters >> interval: 300 >> meters: >> - cpu >> - cpu_l3_cache >> - memory.usage >> - network.incoming.bytes >> - network.incoming.packets >> - network.outgoing.bytes >> - network.outgoing.packets >> - disk.device.read.bytes >> - disk.device.read.requests >> - disk.device.write.bytes >> - disk.device.write.requests >> - hardware.cpu.util >> - hardware.memory.used >> - hardware.memory.total >> - hardware.memory.buffer >> - hardware.memory.cached >> - hardware.memory.swap.avail >> - hardware.memory.swap.total >> - hardware.system_stats.io.outgoing.blocks >> - hardware.system_stats.io.incoming.blocks >> - hardware.network.ip.incoming.datagrams >> - hardware.network.ip.outgoing.datagrams >> > > Here is my ceilometer-rootwrap: > >> # Configuration for ceilometer-rootwrap >> # This file should be owned by (and only-writeable by) the root user >> >> [DEFAULT] >> # List of directories to load filter definitions from (separated by ','). >> # These directories MUST all be only writeable by root ! >> filters_path=/etc/ceilometer/rootwrap.d,/usr/share/ceilometer/rootwrap >> >> # List of directories to search executables in, in case filters do not >> # explicitely specify a full path (separated by ',') >> # If not specified, defaults to system PATH environment variable. >> # These directories MUST all be only writeable by root ! >> exec_dirs=/sbin,/usr/sbin,/bin,/usr/bin,/usr/local/sbin,/usr/local/bin >> >> # Enable logging to syslog >> # Default value is False >> use_syslog=False >> >> # Which syslog facility to use. >> # Valid values include auth, authpriv, syslog, user0, user1... >> # Default value is 'syslog' >> syslog_log_facility=syslog >> >> # Which messages to log. >> # INFO means log all usage >> # ERROR means only log unsuccessful attempts >> syslog_log_level=ERROR >> > > What configuration is wrong ? > > Le ven. 3 juil. 2020 à 13:53, Matthias Runge <mrunge(a)matthias-runge.de> a > écrit : > >> Okay, that doesn't really help with debugging though. >> >> Method not allowed is returned eg. when the endpoint expected an http >> push where your browser did an http get (that's correct). >> >> What I'd do next is to configure ceilometer to send to a different http >> endpoint (like a webserver on your workstation, just for debugging >> purposes). >> >> Verify that the push gateway works as expected, >> https://github.com/prometheus/pushgateway >> has some curl commands mentioned for debugging purposes. >> >> >> Matthias >> >> On 03/07/2020 13:07, Samuel Mutel wrote: >> > If I go to http://10.60.4.11:9091/metrics/job/ceilometer with the web >> > browser I receive: Method Not Allowed but i think it's normal. >> > http://10.60.4.11:9091/metrics is working with metrics. >> > >> > The pushgateway and the ceilometer is working on the same host for my >> > test so no network/firewall issue. >> > >> > Logs of the pushgateway is only these ones: >> > level=info ts=2020-07-03T11:04:35.907Z caller=main.go:83 msg="starting >> > pushgateway" version="(version=1.2.0, branch=HEAD, >> > revision=b7e0167e9574f4f88404dde9653ee1d3c940f2eb)" >> > level=info ts=2020-07-03T11:04:35.908Z caller=main.go:84 >> > build_context="(go=go1.13.8, user=root@0e823ccfff84, >> > date=20200311-18:51:01)" >> > level=info ts=2020-07-03T11:04:35.911Z caller=main.go:137 >> > listen_address=:9091 >> > >> > Le ven. 3 juil. 2020 à 12:14, Matthias Runge <mrunge(a)matthias-runge.de >> > <mailto:mrunge@matthias-runge.de>> a écrit : >> > >> > On 03/07/2020 11:25, Samuel Mutel wrote: >> > > Hello, >> > > >> > > I have two questions about ceilometer (openstack version rocky). >> > > >> > > * First of all, it seems that ceilometer is sending metrics >> > every hour >> > > and I don't understand why. >> > > * Next, I am not able to setup ceilometer to send metrics to >> > > prometheus pushgateway. >> > > >> > > Here is my configuration: >> > > >> > > sources: >> > > - name: meter_file >> > > interval: 30 >> > > meters: >> > > - "*" >> > > sinks: >> > > - prometheus >> > > >> > > sinks: >> > > - name: prometheus >> > > publishers: >> > > - >> > prometheus://10.60.4.11:9091/metrics/job/ceilometer >> > <http://10.60.4.11:9091/metrics/job/ceilometer> >> > > <http://10.60.4.11:9091/metrics/job/ceilometer> >> > > >> > > >> > > Here is the error I received: >> > > >> > > vcpus{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"} 2 >> > > # TYPE memory gauge >> > > memory{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"} >> 2048 >> > > # TYPE disk.ephemeral.size gauge >> > > >> > >> disk.ephemeral.size{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"} >> > > 0 >> > > # TYPE disk.root.size gauge >> > > >> > disk.root.size{resource_id="7fab268b-ca7c-4692-a103-af4a69f817e4"} >> 0 >> > > : HTTPError: 400 Client Error: Bad Request for url: >> > > http://10.60.4.11:9091/metrics/job/ceilometer >> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http >> > > Traceback (most recent call last): >> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http >> > File >> > > >> "/usr/lib/python2.7/dist-packages/ceilometer/publisher/http.py", >> > > line 178, in _do_post >> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http >> >> > > res.raise_for_status() >> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http >> > File >> > > "/usr/lib/python2.7/dist-packages/requests/models.py", line >> > 935, in >> > > raise_for_status >> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http >> >> > > raise HTTPError(http_error_msg, response=self) >> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http >> > > HTTPError: 400 Client Error: Bad Request for url: >> > > http://10.60.4.11:9091/metrics/job/ceilometer >> > > 2020-07-01 17:00:12.272 11375 ERROR ceilometer.publisher.http >> > > >> > > >> > > Thanks for your help on this topic. >> > >> > >> > Hi, >> > >> > first obvious question: >> > >> > are you sure that there is something listening under >> > http://10.60.4.11:9091/metrics/job/ceilometer ? >> > >> > Would you have some error logs from the other side? It seems that >> > ceilometer is trying to dispatch as expected. >> > >> > Matthias >> > >> >>

1 0

[neutron] Implementing BGP over network:routed for IPv6 in Neutron, with DVR capabilities
by Thomas Goirand 14 Aug '20

14 Aug '20

Hi, When these patches are approved: https://review.opendev.org/486450 https://review.opendev.org/669395 we will effectively have BGP announcing for floating IPs and router gateways, with a provider network as next BGP HOP. I tested in experimentally, and it does work. There's more work to be done on it to make it better (like, eliminating GARP requests and getting neutron-dynamic-routing to know when a floating moves from one segment to another), as seen in the commends of #669395, but it works. Now, I'd like to have the same feature for IPv6. Having a segmented IPv6 L2 network already works, though isn't this always going through the network nodes still? I see no reason why IPv6 would always go through network nodes, and I would like to eliminate this SPOF. Has anyone worked on this? Or is there anyone with some advice on how to start? Is there some blueprints somewhere? I'm not sure what this implies, and where to start my research on this. But I really would love, moving forward, to have such a feature. Would anyone (try to) contribute this with me? Cheers, Thomas Goirand (zigo)

1 0

[tripleo] Specify different interface name in single nic vlans without external network installation
by Marco Marino 14 Aug '20

14 Aug '20

Hi, I'm trying to install openstack using tripleo on preprovisioned servers. My (desired) environment is quite simple: 1 controller and 1 compute node. Here is what I did: - Installed undercloud with 192.168.25.0/24 as a ctlplan subnet. local_ip = 192.168.25.2; undercloud_public_host=192.168.25.4; undercloud_admin_host = 192.168.25.3 - Installed 2 servers (compute and controller) with centos 8. Hardware requirements are satisfied (32 GB of ram, 100GB disk....) - Manually created user stack on compute and controller and added it to the sudoers list. - Manually Installed openstack repositories (sudo -E tripleo-repos -b ussuri current-tripleo-rdo) - Manually installed openstack required openstack packages: sudo yum install python3-heat-agent* -y Now I'd like to use 192.168.25.0/24 as "installation network" (network used by ansible) and I'm trying to configure one single nic with vlans. Please note that on all servers (undercloud included) I have 2 physical interfaces with the same name: enp1s0 and enp7s0. enp1s0 is used for a completely openstack-detached network: 192.168.2.0/24 and enp7s0 is used for 192.168.25.0/24 More precisely: Controller-0 = 192.168.25.10 Compute-0 = 192.168.25.20 Furthermore, I confirm that I can reach <local_ip> from both nodes using ping and curl (on port 8004). And I added 2 lines in /etc/hosts on undercloud: 192.168.25.10 controller-0 192.168.25.20 compute-0 Now I'm really confused about what to do. I tried with: openstack overcloud deploy --templates --disable-validations -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml -e templates/network-environment-overrides.yaml -e templates/ctlplane-assignment.yaml -e templates/nameservers.yaml -e templates/node-info.yaml -e templates/hostnamemap.yaml -n templates/network_data.yaml node-info.yaml: http://paste.openstack.org/show/796840/ hostnamemap.yaml: http://paste.openstack.org/show/796841/ network-environment-overrides.yaml: http://paste.openstack.org/show/796842/ ctlplane-assignment.yaml: http://paste.openstack.org/show/796843/ nameservers.yaml: http://paste.openstack.org/show/796844/ network_data.yaml: http://paste.openstack.org/show/796845/ How can I specify nic2 for single nic vlans without external network? Please can you provide an example with "openstack overcloud deploy" complete command? I'm reading documentation but I do not understand how to do and it's really frustrating. Thank you, Marco

1 0

Openstack-Train VCPU issue in Hyper-V
by its-openstack＠zohocorp.com 14 Aug '20

14 Aug '20

Dear Team, We are using Openstack-Train in our organization.We have created windows server 2016 Std R2 instances with this flavor m5.xlarge ( RAM - 65536 , Disk - 500 , VCPUs - 16 ).Once Hyper-V future enabled in this instances VCPU count is automatically reduced to 1 core after restart.Even we have enabled nested virtualisation in openstack compute server.Please help us to short out this issue. #cat /sys/module/kvm_intel/parameters/nested Y Regards, Sysadmin.

2 1

[masakari] Meetings
by Radosław Piliszek 14 Aug '20

14 Aug '20

Hello Fellow cloud-HA-seekers, I wanted to attend Masakari meetings but I found the current schedule unfit. Is there a chance to change the schedule? The day is fine but a shift by +3 hours would be nice. Anyhow, I wanted to discuss [1]. I've already proposed a change implementing it and looking forward to positive reviews. :-) That said, please reply on the change directly, or mail me or catch me on IRC, whichever option sounds best to you. [1] https://blueprints.launchpad.net/masakari/+spec/customisable-ha-enabled-ins… -yoctozepto

2 2

[Ocata][Heat] Strange error returned after stack creation failure -r aw template with id xxx not found
by Laurent Dumont 14 Aug '20

14 Aug '20

Hi! We are currently troubleshooting a Heat stack issue where one of the stack (one of 25 or so) is failing to be created properly (seemingly randomly). The actual error returned by Heat is quite strange and Google has been quite sparse in terms of references. The actual error looks like the following (I've sanitized some of the names): Resource CREATE failed: resources.potato: Resource CREATE failed: resources[0]: raw template with id 22273 not found heat resource-list STACK_NAME_HERE -n 50 > > +------------------+--------------------------------------+-------------------------+-----------------+----------------------+--------------------------------------------------------------------------+ > | resource_name | physical_resource_id | resource_type > | resource_status | updated_time | stack_name > | > > +------------------+--------------------------------------+-------------------------+-----------------+----------------------+--------------------------------------------------------------------------+ > | potato | RESOURCE_ID_HERE | OS::Heat::ResourceGroup | > CREATE_FAILED | 2020-07-18 T19:52:10Z | nested_stack_1_STACK_NAME_HERE > | > | potato_server_group | RESOURCE_ID_HERE | OS::Nova::ServerGroup | > CREATE_COMPLETE | 2020-07-21T19:52:10Z | nested_stack_1_STACK_NAME_HERE > | > | 0 | | potato1.yaml > | CREATE_FAILED | 2020-07-18T19:52:12Z | nested_stack_2_STACK_NAME_HERE > | > | 1 | | potato1.yaml > | INIT_COMPLETE | 2020-07- 18 T19:52:12Z | > nested_stack_2_STACK_NAME_HERE | > > +------------------+--------------------------------------+-------------------------+-----------------+----------------------+--------------------------------------------------------------------------+ > The template itself is pretty simple and attempts to create a ServerGroup and 2 VMs (as part of the ResourceGroup). My feeling is that one the creation of those machines fails and Heat get's a little cooky and returns an error that might not be the actual root cause. I would have expected the VM to show up in the resource list but I just see the source "yaml". Has anyone seen something similar in the past? Thanks!

2 3

[cinder] driver maintainers: 3rd party CI checkpoint reminder
by Brian Rosmaita 14 Aug '20

14 Aug '20

Hello Cinder driver maintainers, Around the time of the Feature Freeze (10 September), the Cinder team will be looking at the Third Party CIs to assess compliance [0]. Out of compliance drivers will be marked as 'unsupported' in the Victoria release. We can avoid a lot of unpleasantness if you take this opportunity to review the current situation of your driver's CI [1] and, if necessary, take appropriate steps to get it back into compliance before 10 September. cheers, brian [0] https://docs.openstack.org/cinder/latest/drivers-all-about.html#driver-comp… [1] http://cinderstats.ivehearditbothways.com/cireport.txt

1 0

[cinder] driver features declaration for victoria next week
by Brian Rosmaita 14 Aug '20

14 Aug '20

Hello Cinder driver maintainers, This is a reminder that new features added to Cinder drivers for the Victoria release must be merged at the time of the OpenStack-wide Feature Freeze, which is coming up soon (10 September, to be specific). In order to avoid the Festival of Insane Driver Reviewing that we had last cycle, if you have un-merged driver features that you would like to land in Victoria, please post a blueprint in Launchpad listing the Gerrit reviews of the associated patches before the next Cinder weekly meeting (that is, before 19 August at 1400 UTC). This will help the team prioritize reviews and give you candid early feedback on whether the features look ready. You can look among the Ussuri blueprints for examples; contact me in IRC if you have any questions. Due to the 7 September holiday in the USA, there will be reduced reviewing bandwidth right around the Feature Freeze, so that's why I'm asking you to plan ahead. cheers, brian

1 0

[cinder] victoria new feature status checkpoint next week
by Brian Rosmaita 14 Aug '20

14 Aug '20

If you are working on a Cinder feature for Victoria that hasn't merged yet, please add it to the agenda for next week's cinder weekly meeting on 19 August at 1400 UTC: https://etherpad.opendev.org/p/cinder-victoria-meetings If your feature requires client support, keep in mind that the final release for client libraries is in four weeks. Any client changes must be reviewed, tested, and merged before 10 September. Keep in mind that 7 September is a holiday for many Cinder core reviewers, so it is likely that we will have reduced reviewer bandwidth around the time of the Feature Freeze. So please plan ahead. cheers, brian

1 0

[cinder] victoria os-brick release coming soon
by Brian Rosmaita 13 Aug '20

13 Aug '20

Just a quick reminder that the victoria os-brick release is 3 weeks away. Reviews may be a bit slower than usual given that people may be taking some end-of-the-summer vacation, so if you have an important patch for os-brick, please take the initiative to raise awareness in the #openstack-cinder IRC channel if it's not getting the attention it deserves. cheers, brian

1 0