May 2024 - openstack-discuss - lists.openstack.org

Antelope Magnum: creating a new cluster makes the/some others unhealthy
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I have been playing with Magnum (Antelope), creating a lot of clusters with the same template and varying the number of masters and/or nodes, having ~10 clusters started in the same project. I observed recently that cluster configuration that used to work well were not working any more (CREATE_FAILED during the master deployment in general). I have not found any evidence of the cause digging in various logs but I observed that with the current list of active clusters, if add a new one (even with a configuration as minimal as 1 master and 1 node), not only its creation fails but also the other (or at least) running clusters become unhealthy. If I delete the failed cluster, the other clusters return to the healthy state. It seems very reproducible (I did it more than 10 times) but I still don't see any message in the logs that could help to identify the cause. I've the feeling (but may be wrong) that it is related to either an insufficient project quota or an insufficient limit on one of the OpenStack service. Any idea on a possible cause or any advice on where to look for some information? Thanks in advance. Michel

2 6

[all] SQLAlchemy minimum requirement
by Pierre Riteau 03 Jun '24

03 Jun '24

Hello everyone, Historically, many projects were using the following requirement for SQLAlchemy: SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10 These banned versions were present in global-requirements.txt until earlier this month [1]. The remaining projects with the above requirement must change it to fix their requirements-check job (ERROR: Requirement for package SQLAlchemy excludes a version not excluded in the global list). While fixing the requirement for blazar, I realised that OpenStack projects are using a wide range of minimum requirements. For example, networking-sfc had already bumped the minimum to SQLAlchemy 2.0 [2], although this is being reverted [3]. While all projects must be compatible with the upper constraint (currently 2.0.30), should we agree on a maximum version for the lower bound? The 2024.2 release targets Ubuntu Jammy, which includes python3-sqlalchemy 1.4.31, and Debian bookworm, which includes python3-sqlalchemy 1.4.46. Thanks, Pierre Riteau (priteau) [1] https://review.opendev.org/c/openstack/requirements/+/879743 [2] https://review.opendev.org/c/openstack/networking-sfc/+/920046 [3] https://review.opendev.org/c/openstack/networking-sfc/+/920868

2 1

The outbound unicast traffic of VMS with security group enabled is broadcast
by renwc＠chinatelecom.cn 31 May '24

31 May '24

hello everyone， Here is a description of a bug I encountered using openstack Security Group,If security groups are enabled for virtual nics on cloud hosts, The outgoing traffic of the VMS enabled in the security group should be sent in unicast mode, but it is eventually forwarded to all ports in the vlan on the ovs br-int bridge. Does the openstack open source community have a patch to fix this bug? Can you provide patch? The following is my debugging process in detail. The unicast packet sent by the remote virtual machine 172.83.160.63 to the local virtual machine 172.83.160.74 is passed to the br-ex->br-int-> VM, Because the security group is enabled on the VM NIC, The br-int cannot learn the entry whose port is patch-ex and mac address is vm2. Therefore those packets sent from vm1 to vm2 are broadcast in vlans because they fail to match mac address entries. 172.83.160.74 patch-int 172.83.160.63 vm1 --- ovs(br-int) ------ ovs(br-ex)--- tor(physical switch) ......vm2 patch-ex The following is the ovs kernel flow table. The mac address of vm1 is fa:16:3e:6c:f3:09,The mac address of vm2 is fa:16:3e:d2:14:b5, the action of the flow table is multiple ports. recirc_id(0x314123),in_port(8),ct_state(-new+est-rel-rpl),eth(src=fa:16:3e:d2:14:b5,dst=fa:16:3e:6c:f3:09),eth_type(0x0800),ipv4(proto=6,frag=no),tcp(dst=2048/0xf800), packets:60100, bytes:14257193, used:2.035s, flags:P., actions:push_vlan(vid=5,pcp=0),3,pop_vlan,push_vlan(vid=1732,pcp=0),1,pop_vlan,7,9,10,15,5,18 recirc_id(0x314123),in_port(8),ct_state(-new+est-rel+rpl-inv+trk),ct_zone(0x5),ct_mark(0),eth(src=fa:16:3e:d2:14:b5,dst=fa:16:3e:6c:f3:09),eth_type(0x0800),ipv4(frag=no), packets:1, bytes:54, used:2.037s, flags:., actions:push_vlan(vid=5,pcp=0),3,pop_vlan,push_vlan(vid=1732,pcp=0),1,pop_vlan,7,9,10,15,5,18 After careful debugging analysis, the openflow action in the flow table 82 added by neutron is the VM port instead of normal. As a result, the ovs br-int bridge cannot learn the mac entry of vm2. table=82, priority=77,ct_state=+new-est,tcp,reg5=0x9,tp_dst=0x4/0xfffc actions=ct(commit,zone=NXM_NX_REG6[0..15]),output:9 table=82, priority=77,ct_state=+est-rel-rpl,tcp,reg5=0x9,tp_dst=0x4/0xfffc actions=output:9 Looking forward to reply thanks. weichun ren

2 1

The outgoing unicast traffic from the VMS enabled in the security group is eventually forwarded to all ports.
by renwc＠chinatelecom.cn 31 May '24

31 May '24

Hello， Here is a description of a bug I encountered using openstack Security Group,If security groups are enabled for virtual nics on VMS, The outgoing traffic from the VMS enabled in the security group should be sent in unicast mode, but it is eventually forwarded to all ports in the vlan on the ovs br-int bridge. Does the openstack open source community have a patch to fix this bug? Can you provide patch fixed the issue? The following is my debugging process in detail. The unicast packet sent by the remote virtual machine 172.83.160.63 to the local virtual machine 172.83.160.74 is passed to the br-ex->br-int-> VM, Because the security group is enabled on the VM NIC, The br-int cannot learn the entry whose port is patch-ex and mac address is vm2. Therefore those packets sent from vm1 to vm2 are broadcast in vlans because they fail to match mac address entries. 172.83.160.74 patch-int 172.83.160.63 vm1 --- ovs(br-int) ------ ovs(br-ex)--- tor(physical switch) ......vm2 patch-ex The following is the ovs kernel flow table. The mac address of vm1 is fa:16:3e:6c:f3:09,The mac address of vm2 is fa:16:3e:d2:14:b5, the action of the flow table is multiple ports. recirc_id(0x314123),in_port(8),ct_state(-new+est-rel-rpl),eth(src=fa:16:3e:d2:14:b5,dst=fa:16:3e:6c:f3:09),eth_type(0x0800),ipv4(proto=6,frag=no),tcp(dst=2048/0xf800), packets:60100, bytes:14257193, used:2.035s, flags:P., actions:push_vlan(vid=5,pcp=0),3,pop_vlan,push_vlan(vid=1732,pcp=0),1,pop_vlan,7,9,10,15,5,18 recirc_id(0x314123),in_port(8),ct_state(-new+est-rel+rpl-inv+trk),ct_zone(0x5),ct_mark(0),eth(src=fa:16:3e:d2:14:b5,dst=fa:16:3e:6c:f3:09),eth_type(0x0800),ipv4(frag=no), packets:1, bytes:54, used:2.037s, flags:., actions:push_vlan(vid=5,pcp=0),3,pop_vlan,push_vlan(vid=1732,pcp=0),1,pop_vlan,7,9,10,15,5,18 After careful debugging analysis, the openflow action in the flow table 82 added by neutron is the VM port instead of normal. As a result, the ovs br-int bridge cannot learn the mac entry of vm2. table=82, priority=77,ct_state=+new-est,tcp,reg5=0x9,tp_dst=0x4/0xfffc actions=ct(commit,zone=NXM_NX_REG6[0..15]),output:9 table=82, priority=77,ct_state=+est-rel-rpl,tcp,reg5=0x9,tp_dst=0x4/0xfffc actions=output:9 Was the issue above fixed in the openstack ? Looking forward to reply thanks. Best Regards weichun ren

1 0

Re: [tc][all][senlin] Retiring the Senlin project
by Ghanshyam Mann 30 May '24

30 May '24

---- On Thu, 30 May 2024 11:43:00 -0700 Sang Tran Quoc wrote --- > > Dear Ghanshyam, > > Greeting! I’m Sang from Viet Nam and now I’m working at the FPT Smart Cloud (Silver member of OpenInfra) as the openstack team leader. We have teammate with about 10 members who maintains the senlin project on our production AutoScaling Feature (my presentation about senlin at last VietOpenInfra summithttps://www.youtube.com/watch?v=aYXDI4mwohQ&t=1s) It’s sad to hear about community retired this project from upstream so I would like to nominate our team to continue maintain this project and I will handle the PTL role for further roadmap, bugs fix and feature. > > Hopefully it’s not late, I’m look forward to hearing from you for further discuss about next chance for senlin project. ++ openstack-discuss ML Hi Sang, Thanks for reaching out and its good to hear about Senlin project usage and your interest in maintaining it. > Hopefully it’s not late, I’m look forward to hearing from you for further discuss about next chance for senlin project. Unfortunately, TC has already retired the Senlin project[1]. Since last cycle, we have been waiting for interested maintainers to step up. * No Leader in 2024.1 cycle ** https://etherpad.opendev.org/p/2024.1-leaderless * After TC discussion, Senlin has been marked as an 'inactive' project since '2024.1' cycle ** https://review.opendev.org/c/openstack/governance/+/902626 ** This was the time we were expecting someone to come forward and step up to maintain it. * No leader for Senlin in 2024.2 cycles also ** https://etherpad.opendev.org/p/2024.2-leaderless * Considering no leader for two cycles, no maintainer, and the process, TC initiated the retirement: ** https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… But this is not the end and Senlin can still re-become the official OpenStack project[2]. For that, you need to re-apply Senlin as a new project in OpenStack, please check the process: - https://governance.openstack.org/tc/reference/new-projects-requirements.html [1] - https://review.opendev.org/c/openstack/governance/+/919347 - https://review.opendev.org/q/topic:%22retire-senlin%22+ [2]https://governance.openstack.org/tc/reference/dropping-projects.html#re-b… -gmann > > > Thanks & Best Regards > > > Sang Tran Quoc (Mr.) > FPT Smart Cloud > FPT Tower, 10 Pham Van Bach Str., Cau Giay Dist., Ha Noi, Viet Nam > E: sangtq8(a)fpt.com > T: 0962949843 > W: https://fptcloud.com | https://fpt.ai > > > > > From:Ghanshyam Mann gmann(a)ghanshyammann.com> > Date: Saturday, 11 May 2024 at 09:06 > To: openstack-discuss openstack-discuss(a)lists.openstack.org> > Subject: Re: [tc][all][senlin] Retiring the Senlin project > ---- On Wed, 24 Apr 2024 13:39:33 -0700 Ghanshyam Mann wrote --- > > Hello everyone, > > > > As you might know, the Senlin project is not active and was marked 'inactive' > > since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer > > stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next > > step is to retire the project. > > > > I'm sending this email to propose the Senlin retirement and based on the response, > > I will proceed next. > > Seeing no response to this email or no volunteer to maintain this project, I have proposed > the retirement in governance: > > - https://review.opendev.org/c/openstack/governance/+/919347 > > -gmann > > > > > [1] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… > > [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72 > > > > -gmann > > >

1 0

[neutron][all] OOM killer on coverage job
by Brian Haley 30 May '24

30 May '24

Hi, Neutron has been having issues with our coverage gate job triggering the OOM killer since last week [0], which I just confirmed by holding a node and looking in the logs. It started happening after the sqlalchemy 2.0 bump [1], but that just might be exposing the underlying issue. Running locally I can see via /proc/meminfo that memory is getting consumed: MemTotal: 8123628 kB MemFree: 1108404 kB And via ps it's the coverage processes doing it: PID %MEM RSS PPID TIME NLWP WCHAN COMMAND 4315 30.9 2516348 4314 01:29:07 1 - /opt/stack/neutron/.tox/cover/bin/python /opt/stack/neutron/.tox/cover/bin/coverage run --source neutron --parallel-mode -m stestr.subunit_runner.run discover -t ./ ./neutron/tests/unit --load-list /tmp/tmp0rhqfwhz 4313 30.0 2437500 4312 01:28:50 1 - /opt/stack/neutron/.tox/cover/bin/python /opt/stack/neutron/.tox/cover/bin/coverage run --source neutron --parallel-mode -m stestr.subunit_runner.run discover -t ./ ./neutron/tests/unit --load-list /tmp/tmpfzmqyuub (and the test hasn't even finished yet) Only workaround seems to be reducing concurrency [2]. Have any other projects seen anything similar? (and sorry for the html email) -Brian [0] https://bugs.launchpad.net/neutron/+bug/2065821 [1] https://review.opendev.org/c/openstack/requirements/+/879743 [2] https://review.opendev.org/c/openstack/neutron/+/920766

4 6

A bug about openstack security groups
by renwc＠chinatelecom.cn 30 May '24

30 May '24

hello everyone， Here is a description of a bug I encountered using openstack Security Group,If security groups are enabled for virtual nics on cloud hosts, The outgoing traffic of the VMS enabled in the security group should be sent in unicast mode, but it is eventually forwarded to all ports in the vlan on the ovs br-int bridge. Does the openstack open source community have a patch to fix this bug? Can you provide patch? The following is my debugging process in detail. The unicast packet sent by the remote virtual machine 172.83.160.63 to the local virtual machine 172.83.160.74 is passed to the br-ex->br-int-> VM, Because the security group is enabled on the VM NIC, The br-int cannot learn the entry whose port is patch-ex and mac address is vm2. Therefore those packets sent from vm1 to vm2 are broadcast in vlans because they fail to match mac address entries. 172.83.160.74 patch-int 172.83.160.63 vm1 --- ovs(br-int) ------ ovs(br-ex)--- tor(physical switch) ......vm2 patch-ex The following is the ovs kernel flow table. The mac address of vm1 is fa:16:3e:6c:f3:09,The mac address of vm2 is fa:16:3e:d2:14:b5, the action of the flow table is multiple ports. recirc_id(0x314123),in_port(8),ct_state(-new+est-rel-rpl),eth(src=fa:16:3e:d2:14:b5,dst=fa:16:3e:6c:f3:09),eth_type(0x0800),ipv4(proto=6,frag=no),tcp(dst=2048/0xf800), packets:60100, bytes:14257193, used:2.035s, flags:P., actions:push_vlan(vid=5,pcp=0),3,pop_vlan,push_vlan(vid=1732,pcp=0),1,pop_vlan,7,9,10,15,5,18 recirc_id(0x314123),in_port(8),ct_state(-new+est-rel+rpl-inv+trk),ct_zone(0x5),ct_mark(0),eth(src=fa:16:3e:d2:14:b5,dst=fa:16:3e:6c:f3:09),eth_type(0x0800),ipv4(frag=no), packets:1, bytes:54, used:2.037s, flags:., actions:push_vlan(vid=5,pcp=0),3,pop_vlan,push_vlan(vid=1732,pcp=0),1,pop_vlan,7,9,10,15,5,18 After careful debugging analysis, the openflow action in the flow table 82 added by neutron is the VM port instead of normal. As a result, the ovs br-int bridge cannot learn the mac entry of vm2. table=82, priority=77,ct_state=+new-est,tcp,reg5=0x9,tp_dst=0x4/0xfffc actions=ct(commit,zone=NXM_NX_REG6[0..15]),output:9 table=82, priority=77,ct_state=+est-rel-rpl,tcp,reg5=0x9,tp_dst=0x4/0xfffc actions=output:9 Looking forward to reply thanks. 任维春智能计算产品部/智能云网业务部邮箱：renwc(a)chinatelecom.cn 地址：北京市西城区西直门内大街118-1 中电信数智科技有限公司

1 0

[Diskimage-builder]Supported architectures for creating OS images in Diskimage-builder
by Junya Noguchi (Fujitsu) 29 May '24

29 May '24

To whom it may concern I am investigating which Linux distributions and versions for aarch64 are supported by Diskimage-builder. I found that many newer versions of Linux distributions are supported*, but there is no information about supported architectures in the commit messages. * References: Ubuntu: https://review.opendev.org/c/openstack/project-config/+/916050 RHEL: https://review.opendev.org/c/openstack/diskimage-builder/+/877347 SLES: https://review.opendev.org/c/openstack/diskimage-builder/+/619186 SUSE: https://review.opendev.org/c/openstack/diskimage-builder/+/908421 CentOS: https://review.opendev.org/c/openstack/tripleo-quickstart/+/843102 Does Diskimage-builder support building aarch64 images for those distributions? If not, a list of supported distros for aarch64, would be helpful. Best regards, Junya Noguchi.

2 4

[kolla] cancelling weekly meeting today (29th May 2024)
by Michał Nasiadka 29 May '24

29 May '24

Hello Koalas, We’re cancelling the weekly meeting today (29th May 2024) due to no show of majority key members. Let’s meet next week! Thanks for understanding, Michal

1 0

overcloud deploy fails in step 1 | tripleo | wallaby
by Swogat Pradhan 29 May '24

29 May '24

Hi, I am trying to run a stack update and am executing openstack overcloud deploy, but after reaching step 1 and prefetching the container images the deployment fails with following error: s 2024-05-16 19:58:54.138764 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-backup:latest 2024-05-16 19:58:54.143338 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:18.464385 | 43.62s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' s/_base.py\", line 384, in __get_result\n raise self._exception\n File \"/usr/lib64/python3.6/concurrent/futures/thread.py\", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 332, in run_container\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 318, in manage_container\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 1659, in execute\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 1572, in make_started\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 1351, in different\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 692, in __init__\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 699, in defaultize\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee st2024-05-16 19:58:55.487682 | 525400ff-fd40-2d4d-b69c-000000010b27 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 391, in <module>\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 387, in main\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 132, in __init__\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 370, in run\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 337, in check_failures\n File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 586, in result_iterator\n yield fs.pop().result()\n File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 425, in result\n return self.__get_result()\n File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 384, in __get_result\n raise self._exception\n File \"/usr/lib64/python3.6/concurrent/futures/thread.py\", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 332, in run_container\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 318, in manage_container\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 1659, in execute\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 1572, in make_started\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 1351, in different\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 692, in __init__\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 699, in defaultize\n File \"/tmp/ansible_tripleo_container_manage_payload_sg7woby1/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} 2024-05-16 19:58:55.488510 | 525400ff-fd40-2d4d-b69c-000000010b27 | TIMING | Manage Puppet containers (generate config) for step 1 | overcloud-novacompute-3 | 0:23:19.809572 | 82.91s 2024-05-16 19:58:55.859707 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-cinder-volume:latest 2024-05-16 19:58:55.864370 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:20.185418 | 45.34s 2024-05-16 19:58:57.659959 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-iscsid:latest 2024-05-16 19:58:57.664429 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:21.985477 | 47.14s 2024-05-16 19:59:00.211286 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-manila-share:latest 2024-05-16 19:59:00.216236 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:24.537284 | 49.69s 2024-05-16 19:59:02.113037 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-evaluator:latest 2024-05-16 19:59:02.117498 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:26.438544 | 51.59s 2024-05-16 19:59:03.853383 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-listener:latest 2024-05-16 19:59:03.858232 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:28.179287 | 53.33s 2024-05-16 19:59:05.665169 | 525400ff-fd40-2d4d-b69c-000000010c98 | OK | Pre-fetch all the containers | overcloud-controller-1 | item=172.25.201.68:8787/tripleomaster/openstack-aodh-notifier:latest 2024-05-16 19:59:05.670233 | 525400ff-fd40-2d4d-b69c-000000010c98 | TIMING | Pre-fetch all the containers | overcloud-controller-1 | 0:23:29.991278 | 55.14s An exception occurred during task execution. To see the full traceback, use -vvv. The error was: KeyError: 'config' /futures/_base.py\", line 384, in __get_result\n raise self._exception\n File \"/usr/lib64/python3.6/concurrent/futures/thread.py\", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 332, in run_container\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 318, in manage_container\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 1659, in execute\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 1572, in make_started\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 1351, in different\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 692, in __init__\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 699, in defaultize\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\2024-05-16 19:59:06.648301 | 525400ff-fd40-2d4d-b69c-000000010fc3 | FATAL | Manage Puppet containers (generate config) for step 1 | overcloud-controller-no-ceph-3 | error={"changed": false, "module_stderr": "Traceback (most recent call last):\n File \"<stdin>\", line 102, in <module>\n File \"<stdin>\", line 94, in _ansiballz_main\n File \"<stdin>\", line 40, in invoke_module\n File \"/usr/lib64/python3.6/runpy.py\", line 205, in run_module\n return _run_module_code(code, init_globals, run_name, mod_spec)\n File \"/usr/lib64/python3.6/runpy.py\", line 96, in _run_module_code\n mod_name, mod_spec, pkg_name, script_name)\n File \"/usr/lib64/python3.6/runpy.py\", line 85, in _run_code\n exec(code, run_globals)\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 391, in <module>\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 387, in main\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 132, in __init__\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 370, in run\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 337, in check_failures\n File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 586, in result_iterator\n yield fs.pop().result()\n File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 425, in result\n return self.__get_result()\n File \"/usr/lib64/python3.6/concurrent/futures/_base.py\", line 384, in __get_result\n raise self._exception\n File \"/usr/lib64/python3.6/concurrent/futures/thread.py\", line 56, in run\n result = self.fn(*self.args, **self.kwargs)\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 332, in run_container\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible/modules/tripleo_container_manage.py\", line 318, in manage_container\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 1659, in execute\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 1572, in make_started\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 1351, in different\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 692, in __init__\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 699, in defaultize\n File \"/tmp/ansible_tripleo_container_manage_payload_wnnokrwc/ansible_tripleo_container_manage_payload.zip/ansible_collections/containers/podman/plugins/module_utils/podman/podman_container_lib.py\", line 661, in __init__\nKeyError: 'config'\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1} I do not have any idea why this is caused and how to fix it. Can someone please help? With regards, Swogat Pradhan

4 9