November 2022 - openstack-discuss

[Heat] X-OpenStack-Request-ID changed during request processing
by Vadym Markov 07 Nov '22

07 Nov '22

Hi everyone Found that Heat doesn't keep the value of X-OpenStack-Request-ID header during request processing. Simple stack delete leads to 4 different request id, only the last of them corresponds to heat engine processing the request. Investigation shows that req-id "duplicates" twice - when client goes via redirect response returned by API and at version negotiation at every connect to API endpoint. I assume that this situation is not expected, and it really spoils work with heat api logs. Any ideas on how to make the situation more clear? Also looks like that header name is case-dependent in some places, and Oslo-middleware and Heat handles them differently. However, it needs additional investigation. I propose to turn off logging for successful negotiation and re-use req-id in keystoneuth during redirect. If it is ok, I plan to submit corresponding patches to upstream. Here is a piece of log related to stack delete. Only "req-843ac3a4..." exists in the heat-engine log. лис 07 12:21:59 ubuntu devstack(a)h-api.service[172816]: DEBUG heat.api.middleware.version_negotiation [None req-5b613e29-8dd9-458d-b908-4802f90c0b2f admin admin] Processing request: DELETE /heat-api/v1/8a76e6c8cc714288823dfe3677174893/stacks/stack_1 Accept: application/json {{(pid=172816) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:57}} лис 07 12:21:59 ubuntu devstack(a)h-api.service[172816]: DEBUG heat.api.middleware.version_negotiation [None req-5b613e29-8dd9-458d-b908-4802f90c0b2f admin admin] Matched versioned URI. Version: 1.0 {{(pid=172816) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:69}} лис 07 12:21:59 ubuntu devstack(a)h-api.service[172816]: INFO heat.common.wsgi [None req-ec73b5d8-5025-40e0-90d7-4010d39cb450 admin admin] Processing request: DELETE /heat-api/v1/8a76e6c8cc714288823dfe3677174893/stacks/stack_1 лис 07 12:21:59 ubuntu devstack(a)h-api.service[172816]: DEBUG heat.common.wsgi [None req-ec73b5d8-5025-40e0-90d7-4010d39cb450 admin admin] Calling StackController.lookup {{(pid=172816) __call__ /opt/stack/heat/heat/common/wsgi.py:823}} лис 07 12:21:59 ubuntu devstack(a)h-api.service[172816]: [pid: 172816|app: 0|req: 12/24] 192.168.123.5 () {66 vars in 1391 bytes} [Mon Nov 7 12:21:59 2022] DELETE /heat-api/v1/8a76e6c8cc714288823dfe3677174893/stacks/stack_1 => generated 377 bytes in 167 msecs (HTTP/1.1 302) 5 headers in 289 bytes (1 switches on core 3) лис 07 12:21:59 ubuntu devstack(a)h-api.service[172815]: DEBUG heat.api.middleware.version_negotiation [None req-f1eb3c1b-b76c-42fa-8998-31bb2cc43030 admin admin] Processing request: DELETE /heat-api/v1/8a76e6c8cc714288823dfe3677174893/stacks/stack_1/05e56d91-6b11-4e20-83f3-e2923667019b Accept: application/json {{(pid=172815) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:57}} лис 07 12:21:59 ubuntu devstack(a)h-api.service[172815]: DEBUG heat.api.middleware.version_negotiation [None req-f1eb3c1b-b76c-42fa-8998-31bb2cc43030 admin admin] Matched versioned URI. Version: 1.0 {{(pid=172815) process_request /opt/stack/heat/heat/api/middleware/version_negotiation.py:69}} лис 07 12:21:59 ubuntu devstack(a)h-api.service[172815]: INFO heat.common.wsgi [None req-843ac3a4-408d-4473-844f-f4b109ad1725 admin admin] Processing request: DELETE /heat-api/v1/8a76e6c8cc714288823dfe3677174893/stacks/stack_1/05e56d91-6b11-4e20-83f3-e2923667019b лис 07 12:21:59 ubuntu devstack(a)h-api.service[172815]: DEBUG heat.common.wsgi [None req-843ac3a4-408d-4473-844f-f4b109ad1725 admin admin] Calling StackController.delete {{(pid=172815) __call__ /opt/stack/heat/heat/common/wsgi.py:823}} лис 07 12:21:59 ubuntu devstack(a)h-api.service[172815]: [pid: 172815|app: 0|req: 13/25] 192.168.123.5 () {66 vars in 1502 bytes} [Mon Nov 7 12:21:59 2022] DELETE /heat-api/v1/8a76e6c8cc714288823dfe3677174893/stacks/stack_1/05e56d91-6b11-4e20-83f3-e2923667019b => generated 0 bytes in 118 msecs (HTTP/1.1 204) 2 headers in 112 bytes (1 switches on core 0)

1 0

[neutron] Bug Deputy Report October 31 - November 6
by Elvira Garcia Ruiz 07 Nov '22

07 Nov '22

Hi everyone, I was the bug deputy last week. Find the summary below: High: ------- - neutron.privileged.agent.linux.ip_lib.InterfaceOperationNotSupported: Operation not supported on interface https://bugs.launchpad.net/neutron/+bug/1995735 Assigned to Felipe Reyes Fix proposed https://review.opendev.org/c/openstack/neutron/+/863779 - bulk port create: TypeError: Bad prefix type for generating IPv6 address by EUI-64 https://bugs.launchpad.net/neutron/+bug/1995732 Unassigned - OVN: HA chassis group priority is different than gateway chassis priority https://bugs.launchpad.net/neutron/+bug/1995078 I think this is a legit bug. However, a second look by someone else would be appreciated. Unassigned - ORM session: SQL execution without transaction in progress https://bugs.launchpad.net/neutron/+bug/1995738 Assigned to Felipe Reyes Fix proposed https://review.opendev.org/c/openstack/neutron/+/863780 Undecided: --------------- - [ML2/OVN] After upgrading from Xena to Yoga neutron-dhcp-agent is not working for Baremetals https://bugs.launchpad.net/neutron/+bug/1995287 Rodolfo is discussing with the reporter. It has not yet been confirmed to be a bug. - floating ip portforwarding from external not working https://bugs.launchpad.net/neutron/+bug/1995614 Unassigned >From the description this could be legit but I have no experience with the neutron- L3-agent-floatingip, so an experienced opinion would be appreciated on this one. I left a question for the reporter. Kind regards, Elvira

1 0

[Kolla-ansible][Yoga] How to update an Openstack deployment with new containers?
by wodel youchi 07 Nov '22

07 Nov '22

Hi, I followed this link : https://docs.openstack.org/kolla-ansible/yoga/user/operating-kolla.html It says that : This procedure is for upgrading from series to series, not for doing updates within a series. Inside a series, *it is usually sufficient to just update the kolla-ansible package, rebuild (if needed) and pull the images, and run kolla-ansible deploy again. Please follow release notes to check if there are any issues to be aware of.* In my deployment I am using a local registry, so I pulled the new images, tagged them then pushed them into my registry. I can see that the container's images are newer than the ones I had. I then updated my kolla-ansible package using : $ source yogavenv (yogavenv) $ pip install --upgrade git+ https://opendev.org/openstack/kolla-ansible@stable/yoga Finally I launched the deploy, but it seems that nothing has changed. The containers have not been restarted. Any ideas? Is there a way to verify that a container belongs to a particular image build? How can I be sure that my deployment has been updated with new containers? Regards.

2 1

openstack mentee
by Deeghayu Baddegama 07 Nov '22

07 Nov '22

Hi, I hope to use openstack as the platform in one of my projects. I would like to be a mentee to study about openstack. But when I try to sign up as a mentee it shows an error message. The screenshot attached here. Could you please guide me through this? Thank you. Best regards, Deeghayu Baddegama, Sri Lanka

1 0

openstack multi region with internal ip
by Parsa Aminian 06 Nov '22

06 Nov '22

hello Im using kolla-ansible and for security reasons I use invalid ip for main compute ip . my compute have 2 network adapter and first adapter that is for management is not accessible from outside . the problem is I want to add new region in other datacenter and on openstack with multiregion keystone is deploy only on one region not all regions so my computes with invalid ip can not access keystone anymore . is there any solution for this case ?

1 0

[all][tc] What's happening in Technical Committee: summary 2022 Nov 04: Reading: 5 min
by Ghanshyam Mann 05 Nov '22

05 Nov '22

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * We had this week's video meeting on Nov 03. Most of the meeting discussions are summarized in this email. Meeting recording is available @ https://www.youtube.com/watch?v=cjpxiT2Egok and summary logs are available @ https://meetings.opendev.org/meetings/tc/2022/tc.2022-11-03-15.05.log.html * Next TC weekly meeting will be on Nov 9 Wed at 16:00 UTC, please make note of new day/time for TC meetings. Feel free to add the topic to the agenda[1] by Nov 8. 2. What we completed this week: ========================= * Deprecated openstack-ansible rsyslog roles[2] * Completed the user survey TC question updates for 2023 user survey[3] 3. Activities In progress: ================== TC Tracker for 2023.1 cycle --------------------------------- * Current cycle working items and their progress are present in 2023.1 tracker etherpad[4]. Open Reviews ----------------- * Seven open reviews for ongoing activities[5]. Renovate translation SIG i18 ---------------------------------- * SIG i18 user Survey questions are finalized for 2023 user survey * rosmaita is coordinating with Weblate and as per them there shouldn't be any issue in hosting OpenStack translations. He will connect them with SIG i18n to work out details. TC Weekly meeting new day and time -------------------------------------------- We agreed to shift TC weekly meeting on evry wed on 16:00 UTC. This will be effective from next week (next meeting is on Nov 9, 16:00 UTC)[6]. TC Video meeting discussion ---------------------------------- Even though we discussed it many times in past and TC explained about having a monthly Video call and rest three week as IRC meeting, this came up again in yesterday meeting. We discussed both pros and cons of monthly video call but did not conclude anything. JayF is proposed the resolution to discuss it further if we want to continue on this monthly Video call or not[7]. TC chair nomination & election process ----------------------------------------------- We are formalizing the process of TC chair nomination process. Two options are up for the review[8][9]. Fixing Zuul config error ---------------------------- We are tracking this in one of the item in TC tracker, please refer there for the progress[10]. Project updates ------------------- * Add zookeeper role under OpenStack-Ansible governance[11] * Add Skyline repository for OpenStack-Ansible[12] 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[13]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15:00 UTC [14] 3. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. See you all next week in PTG! [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance/+/863076 [3] https://etherpad.opendev.org/p/tc-2023-user-survey-questions [4] https://etherpad.opendev.org/p/tc-zed-tracker [5] https://review.opendev.org/q/projects:openstack/governance+status:open [6] https://meetings.opendev.org/meetings/tc/2022/tc.2022-11-03-15.05.log.html#… [7] https://review.opendev.org/c/openstack/governance/+/863685 [8] https://review.opendev.org/c/openstack/governance/+/862772 [9] https://review.opendev.org/c/openstack/governance/+/862774 [10] https://etherpad.opendev.org/p/zuul-config-error-openstack [11] https://review.opendev.org/c/openstack/governance/+/863161 [12] https://review.opendev.org/c/openstack/governance/+/863166 [13] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [14] http://eavesdrop.openstack.org/#Technical_Committee_Meeting -gmann

1 0

[neutron] Drivers meeting cancelled
by Rodolfo Alonso Hernandez 04 Nov '22

04 Nov '22

Hello Neutrinos: Today's meeting is cancelled due to the lack of agenda. See you next week. Have a nice weekend.

1 0

(openstack-horizon)
by Adivya Singh 04 Nov '22

04 Nov '22

hi Team, I'm facing one issue after the actual outage happened in my openstack env using horizon, The horizon overview page does not happen. i can get in to Stack page directly, But not the overview it just get timed out i restarted the mem cache services to no avail Is there anything else i can try Regards Adivya Singh

1 0

Infrastructure monitoring
by KK CHN 04 Nov '22

04 Nov '22

List,, Looking for an Infrastructure monitoring tool which is in the open source domain without any proprietary license involved for production use. Need to monitor Switches, UTMs/Firewalls, Physical servers, Virtual Machines/Virtual resources, Storage /Storage servers, Applications/daemons running status etc. Complete Infrastructure monitoring and alerts in advance for rectifying possible issues/outages. Regards, Krish

1 0

(no subject)
by Swogat Pradhan 03 Nov '22

03 Nov '22

I have configured a 3 node pcs cluster for openstack. To test the HA, i issue the following commands: iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT && iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT && iptables -A INPUT -p tcp -m state --state NEW -m tcp --dport 5016 -j ACCEPT && iptables -A INPUT -p udp -m state --state NEW -m udp --dport 5016 -j ACCEPT && iptables -A INPUT ! -i lo -j REJECT --reject-with icmp-host-prohibited && iptables -A OUTPUT -p tcp --sport 22 -j ACCEPT && iptables -A OUTPUT -p tcp --sport 5016 -j ACCEPT && iptables -A OUTPUT -p udp --sport 5016 -j ACCEPT && iptables -A OUTPUT ! -o lo -j REJECT --reject-with icmp-host-prohibited When i issue iptables command on 1 node then it is fenced and forced to reboot and cluster works fine. But when i issue this on 2 of the controller nodes the resource bundles fail and doesn't come back up. [root@overcloud-controller-1 ~]# pcs status Cluster name: tripleo_cluster Cluster Summary: * Stack: corosync * Current DC: overcloud-controller-1 (version 2.1.2-4.el8-ada5c3b36e2) - partition WITHOUT quorum * Last updated: Sat Oct 29 03:15:29 2022 * Last change: Sat Oct 29 03:12:26 2022 by root via crm_resource on overcloud-controller-1 * 19 nodes configured * 68 resource instances configured Node List: * Node overcloud-controller-0: UNCLEAN (offline) * Node overcloud-controller-2: UNCLEAN (offline) * Online: [ overcloud-controller-1 ] Full List of Resources: * ip-172.25.201.91 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 (UNCLEAN) * ip-172.25.201.150 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 (UNCLEAN) * ip-172.25.201.206 (ocf::heartbeat:IPaddr2): Stopped * ip-172.25.201.250 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 (UNCLEAN) * ip-172.25.202.50 (ocf::heartbeat:IPaddr2): Stopped * ip-172.25.202.90 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 (UNCLEAN) * Container bundle set: haproxy-bundle [ 172.25.201.68:8787/tripleomaster/openstack-haproxy:pcmklatest]: * haproxy-bundle-podman-0 (ocf::heartbeat:podman): Started overcloud-controller-0 (UNCLEAN) * haproxy-bundle-podman-1 (ocf::heartbeat:podman): Stopped * haproxy-bundle-podman-2 (ocf::heartbeat:podman): Started overcloud-controller-2 (UNCLEAN) * haproxy-bundle-podman-3 (ocf::heartbeat:podman): Stopped * Container bundle set: galera-bundle [ 172.25.201.68:8787/tripleomaster/openstack-mariadb:pcmklatest]: * galera-bundle-0 (ocf::heartbeat:galera): Stopped overcloud-controller-0 (UNCLEAN) * galera-bundle-1 (ocf::heartbeat:galera): Stopped * galera-bundle-2 (ocf::heartbeat:galera): Stopped overcloud-controller-2 (UNCLEAN) * galera-bundle-3 (ocf::heartbeat:galera): Stopped * Container bundle set: redis-bundle [ 172.25.201.68:8787/tripleomaster/openstack-redis:pcmklatest]: * redis-bundle-0 (ocf::heartbeat:redis): Stopped * redis-bundle-1 (ocf::heartbeat:redis): Stopped overcloud-controller-2 (UNCLEAN) * redis-bundle-2 (ocf::heartbeat:redis): Stopped overcloud-controller-0 (UNCLEAN) * redis-bundle-3 (ocf::heartbeat:redis): Stopped * Container bundle set: ovn-dbs-bundle [ 172.25.201.68:8787/tripleomaster/openstack-ovn-northd:pcmklatest]: * ovn-dbs-bundle-0 (ocf::ovn:ovndb-servers): Stopped overcloud-controller-2 (UNCLEAN) * ovn-dbs-bundle-1 (ocf::ovn:ovndb-servers): Stopped overcloud-controller-0 (UNCLEAN) * ovn-dbs-bundle-2 (ocf::ovn:ovndb-servers): Stopped * ovn-dbs-bundle-3 (ocf::ovn:ovndb-servers): Stopped * ip-172.25.201.208 (ocf::heartbeat:IPaddr2): Started overcloud-controller-2 (UNCLEAN) * Container bundle: openstack-cinder-backup [ 172.25.201.68:8787/tripleomaster/openstack-cinder-backup:pcmklatest]: * openstack-cinder-backup-podman-0 (ocf::heartbeat:podman): Started overcloud-controller-0 (UNCLEAN) * Container bundle: openstack-cinder-volume [ 172.25.201.68:8787/tripleomaster/openstack-cinder-volume:pcmklatest]: * openstack-cinder-volume-podman-0 (ocf::heartbeat:podman): Stopped * Container bundle set: rabbitmq-bundle [ 172.25.201.68:8787/tripleomaster/openstack-rabbitmq:pcmklatest]: * rabbitmq-bundle-0 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-2 (UNCLEAN) * rabbitmq-bundle-1 (ocf::heartbeat:rabbitmq-cluster): Stopped overcloud-controller-0 (UNCLEAN) * rabbitmq-bundle-2 (ocf::heartbeat:rabbitmq-cluster): Stopped * rabbitmq-bundle-3 (ocf::heartbeat:rabbitmq-cluster): Stopped * ip-172.25.204.250 (ocf::heartbeat:IPaddr2): Started overcloud-controller-0 (UNCLEAN) * ceph-nfs (systemd:ceph-nfs@pacemaker): Started overcloud-controller-0 (UNCLEAN) * Container bundle: openstack-manila-share [ 172.25.201.68:8787/tripleomaster/openstack-manila-share:pcmklatest]: * openstack-manila-share-podman-0 (ocf::heartbeat:podman): Started overcloud-controller-0 (UNCLEAN) * stonith-fence_ipmilan-48d539a11820 (stonith:fence_ipmilan): Stopped * stonith-fence_ipmilan-48d539a1188c (stonith:fence_ipmilan): Started overcloud-controller-2 (UNCLEAN) * stonith-fence_ipmilan-246e96349068 (stonith:fence_ipmilan): Started overcloud-controller-2 (UNCLEAN) * stonith-fence_ipmilan-246e96348d30 (stonith:fence_ipmilan): Stopped Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled PCS requires more than half the nodes to be alive for the cluster to work. To fix this step I issued a command:*pcs no-quorum-policy=ignore.* And now the PCS cluster keeps on running even when there is no quorum. Now the issue i have is the mariadb-bundle becomes slave and dosen't get promoted to master. Can you please suggest a proper workaround when more than half nodes go down and my cloud will be still running. With regards, Swogat Pradhan

2 2