May 2024 - openstack-discuss - lists.openstack.org

Issue in Stack.sh
by Lohit S 13 Jun '24

13 Jun '24

How to fix this issue while stack.sh is executed +functions-common:_run_under_systemd:1655 sudo systemctl start devstack(a)q-svc.service +functions-common:run_process:1687 time_stop run_process +functions-common:time_stop:2421 local name +functions-common:time_stop:2422 local end_time +functions-common:time_stop:2423 local elapsed_time +functions-common:time_stop:2424 local total +functions-common:time_stop:2425 local start_time +functions-common:time_stop:2427 name=run_process +functions-common:time_stop:2428 start_time=1716869530330 +functions-common:time_stop:2430 [[ -z 1716869530330 ]] ++functions-common:time_stop:2433 date +%s%3N +functions-common:time_stop:2433 end_time=1716869534546 +functions-common:time_stop:2434 elapsed_time=4216 +functions-common:time_stop:2435 total=16163 +functions-common:time_stop:2437 _TIME_START[$name]= +functions-common:time_stop:2438 _TIME_TOTAL[$name]=20379 +lib/neutron:start_neutron_service_and_check:640 neutron_url= http://192.168.122.150:9696/ +lib/neutron:start_neutron_service_and_check:642 is_service_enabled tls-proxy +functions-common:is_service_enabled:2077 return 1 +lib/neutron:start_neutron_service_and_check:646 '[' '!' -z networking ']' +lib/neutron:start_neutron_service_and_check:647 neutron_url= http://192.168.122.150:9696/networking +lib/neutron:start_neutron_service_and_check:649 echo 'Waiting for Neutron to start...' Waiting for Neutron to start... +lib/neutron:start_neutron_service_and_check:651 local 'testcmd=wget --no-proxy -q -O- http://192.168.122.150:9696/networking' +lib/neutron:start_neutron_service_and_check:652 test_with_retry 'wget --no-proxy -q -O- http://192.168.122.150:9696/networking' 'Neutron did not start' 60 +functions-common:test_with_retry:2359 local 'testcmd=wget --no-proxy -q -O- http://192.168.122.150:9696/networking' +functions-common:test_with_retry:2360 local 'failmsg=Neutron did not start' +functions-common:test_with_retry:2361 local until=60 +functions-common:test_with_retry:2362 local sleep=0.5 +functions-common:test_with_retry:2364 time_start test_with_retry +functions-common:time_start:2407 local name=test_with_retry +functions-common:time_start:2408 local start_time= +functions-common:time_start:2409 [[ -n '' ]] ++functions-common:time_start:2412 date +%s%3N +functions-common:time_start:2412 _TIME_START[$name]=1716869535329 +functions-common:test_with_retry:2365 timeout 60 sh -c 'while ! wget --no-proxy -q -O- http://192.168.122.150:9696/networking; do sleep 0.5; done' +functions-common:test_with_retry:2366 die 2366 'Neutron did not start' +functions-common:die:285 local exitcode=0 [Call Trace] ./stack.sh:1322:start_neutron_service_and_check /home/lohit22/opensource/openstack/devstack/Org/devstack/lib/neutron:652:test_with_retry /home/lohit22/opensource/openstack/devstack/Org/devstack/functions-common:2366:die [ERROR] /home/lohit22/opensource/openstack/devstack/Org/devstack/functions-common:2366 Neutron did not start exit_trap: cleaning up child processes Error on exit

2 2

[openstack charm][tc] openstack-charms still need PTL appointment
by Ghanshyam Mann 11 Jun '24

11 Jun '24

Hello Felipe, openstack-charm team, I tried to reach out to you on IRC but could not connect. This is regarding openstack-charm PTL appointment for the 2024.2 cycle. I am not sure if you noticed or not but there are some comments for you to fix on your PTL appointment. Please check the below Gerrit change and fix the same or let us know if there is a change in the plan. - https://review.opendev.org/c/openstack/governance/+/914254 -gmann

2 3

[swift] clean up orphan segment parts
by Rafael Tavares Oliveira 07 Jun '24

07 Jun '24

Hey there! We've been using swift and we came across this bug https://bugs.launchpad.net/swift/+bug/1813202. We decided to work it out and have an ongoing patch to solve this issue alongside ronaldo.maia(a)luizalabs.com, we would like to share and get some feedback before we submit it. To solve this, we took inspiration in obj/expirer.py: In S3 controllers we register task objects in an internal account, each task object points to an orphaned slo object, which can be created in two situations, when a multipart-upload is not completed or when a multipart object is overwritten. These tasks will be checked by the daemon (a new one we made called "s3-slo-expirer") after some time and deleted from the base accounts. Rafael

2 2

[all][tc] Eventlet migration: Call to action
by Jay Faulkner 07 Jun '24

07 Jun '24

Hi all, As you may have read in Goutham's TC summary, the existing eventlet goal[0] is unlikely to gain consensus to merge. This appears to be due to the difference in needs between OpenStack projects, and belief that it's unlikely a single solution would serve them all. Our current situation is: 1) Eventlet usage must be discontinued. The temporary maintainers are just that -- temporary -- and we should not expect eventlet to continue to be a valid path moving forward. 2) We have basic agreement that shared tooling, such as oslo libraries, should not dictate a specific threading model. In lieu of specific technical leadership and guidance from the TC, this leaves it to individual projects to decide what threading model to use instead of eventlet monkey patching. The unmerged spec[0] may be a good starting point if you're wondering where to start. Importantly: do not let this work wait! It is a significant change and will take a long time to complete. Thanks, Jay Faulkner 0: https://review.opendev.org/c/openstack/governance/+/902585

6 7

Antelope Barbican/Keystone: failure to delete secrets (Keystone TrustNotFound)
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I've been validating Yoga -> Antelope on our preprod cloud. It was working successfully but "suddenly" (i.e. without a kwown/identified change!) when deleting Magnum clusters, the deletion of Barbican secrets (after stack deletion) fail. In the Magnum log, I find the following error: ---- Failed to delete trust: keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-a4400250-e8d5-46ec-a28a-dcb97a682512) (traceback skipped for readability) 2024-05-29 10:43:09.750 2078 INFO magnum.common.cert_manager.barbican_cert_manager [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/edd8487a-6cf1-4f4c-9f53-17… from Barbican. 2024-05-29 10:43:09.750 2078 INFO barbicanclient.base [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Calculated Containers uuid ref: containers/edd8487a-6cf1-4f4c-9f53-173dfae7a8b2 2024-05-29 10:43:09.760 2078 ERROR barbicanclient.client [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] 4xx Client error: Not Found: Secrets container not found. 2024-05-29 10:43:09.761 2078 ERROR magnum.common.cert_manager.barbican_cert_manager [None req-94df560b-d6da-4fba-83a9-34e0b08c64c8 - - - - - -] Error recursively deleting certificate container https://os-77023.lal.in2p3.fr:9311/v1/containers/edd8487a-6cf1-4f4c-9f53-17…: barbicanclient.exceptions.HTTPClientError: Not Found: Secrets container not found. ---- In Keystone log, I find with the same timestamp the following error that seems related to the one above: ---- 2024-05-29 10:43:09.690 1713 WARNING keystone.api._shared.authentication [None req-a4400250-e8d5-46ec-a28a-dcb97a682512 - - - - - -] Could not find trust: cf4b1b6082594025a1ff9e48df383566.: keystone.exception.TrustNotFound: Could not find trust: cf4b1b6082594025a1ff9e48df383566. 2024-05-29 10:43:09.693 1713 WARNING keystone.server.flask.application [None req-a4400250-e8d5-46ec-a28a-dcb97a682512 - - - - - -] Authorization failed. The request you have made requires authentication. from 134.158.77.23: keystone.exception.Unauthorized: The request you have made requires authentication. 2024-05-29 10:43:09.746 1723 ERROR keystone.server.flask.application [None req-8c76f304-c2a3-418c-8372-02c706ed356e 44d8cb3192b3485790ababe29991491a - 5b67a8641dd041089e018ceea9884eac - 5b67a8641dd041089e018ceea9884eac -] Could not find user: 527ec32509d849e8a49b41c3dc4c4c32.: keystone.exception.UserNotFound: Could not find user: 527ec32509d849e8a49b41c3dc4c4c32. ---- It is not clear for me if the ERROR is releated to the 2 warnings as the request is not the same but it seems to happen every time this way. In Barbican logs, there is not much things related except a 401 status for the request corresponding ot the TrustNotFound. I'm looking for advices on how to troubleshoot this. In particular, when the trust should be created and whether I should find somewhere a matching error at the time of the trust creation. It seems the error is only affecting deletion but I may be wrong. for the record, when starting Magnum there is an error complaining about /etc/magnum/keystone_auth_default_policy.json being not found but ignored it so far. And in Keystone an error seems to occur at Magnum cluster creation time: --- 2024-05-29 10:17:17.530 1711 WARNING py.warnings [None req-f484cc35-3e54-4d7f-8688-e6442c90344f a22b1199c74f4a819a49f3ac037a5e44 1dd67113c8bb4ef0a31dfca53335ffbd - - defaul t default] /usr/lib/python3.9/site-packages/oslo_policy/policy.py:1129: UserWarning: Policy "identity:list_services": "role:reader and system_scope:all" failed scope check. The token used to make the request was project scoped but the policy requires ['system'] scope. This behavior may change in the future where using the intended scope is re quired ---- Thanks in advance for any hint. Best regards, Michel

2 3

Antelope Magnum: creating a new cluster makes the/some others unhealthy
by Michel Jouvin 05 Jun '24

05 Jun '24

Hi, I have been playing with Magnum (Antelope), creating a lot of clusters with the same template and varying the number of masters and/or nodes, having ~10 clusters started in the same project. I observed recently that cluster configuration that used to work well were not working any more (CREATE_FAILED during the master deployment in general). I have not found any evidence of the cause digging in various logs but I observed that with the current list of active clusters, if add a new one (even with a configuration as minimal as 1 master and 1 node), not only its creation fails but also the other (or at least) running clusters become unhealthy. If I delete the failed cluster, the other clusters return to the healthy state. It seems very reproducible (I did it more than 10 times) but I still don't see any message in the logs that could help to identify the cause. I've the feeling (but may be wrong) that it is related to either an insufficient project quota or an insufficient limit on one of the OpenStack service. Any idea on a possible cause or any advice on where to look for some information? Thanks in advance. Michel

2 6

[all] SQLAlchemy minimum requirement
by Pierre Riteau 02 Jun '24

02 Jun '24

Hello everyone, Historically, many projects were using the following requirement for SQLAlchemy: SQLAlchemy!=1.1.5,!=1.1.6,!=1.1.7,!=1.1.8,>=1.0.10 These banned versions were present in global-requirements.txt until earlier this month [1]. The remaining projects with the above requirement must change it to fix their requirements-check job (ERROR: Requirement for package SQLAlchemy excludes a version not excluded in the global list). While fixing the requirement for blazar, I realised that OpenStack projects are using a wide range of minimum requirements. For example, networking-sfc had already bumped the minimum to SQLAlchemy 2.0 [2], although this is being reverted [3]. While all projects must be compatible with the upper constraint (currently 2.0.30), should we agree on a maximum version for the lower bound? The 2024.2 release targets Ubuntu Jammy, which includes python3-sqlalchemy 1.4.31, and Debian bookworm, which includes python3-sqlalchemy 1.4.46. Thanks, Pierre Riteau (priteau) [1] https://review.opendev.org/c/openstack/requirements/+/879743 [2] https://review.opendev.org/c/openstack/networking-sfc/+/920046 [3] https://review.opendev.org/c/openstack/networking-sfc/+/920868

2 1

The outbound unicast traffic of VMS with security group enabled is broadcast
by renwc＠chinatelecom.cn 31 May '24

31 May '24

hello everyone， Here is a description of a bug I encountered using openstack Security Group,If security groups are enabled for virtual nics on cloud hosts, The outgoing traffic of the VMS enabled in the security group should be sent in unicast mode, but it is eventually forwarded to all ports in the vlan on the ovs br-int bridge. Does the openstack open source community have a patch to fix this bug? Can you provide patch? The following is my debugging process in detail. The unicast packet sent by the remote virtual machine 172.83.160.63 to the local virtual machine 172.83.160.74 is passed to the br-ex->br-int-> VM, Because the security group is enabled on the VM NIC, The br-int cannot learn the entry whose port is patch-ex and mac address is vm2. Therefore those packets sent from vm1 to vm2 are broadcast in vlans because they fail to match mac address entries. 172.83.160.74 patch-int 172.83.160.63 vm1 --- ovs(br-int) ------ ovs(br-ex)--- tor(physical switch) ......vm2 patch-ex The following is the ovs kernel flow table. The mac address of vm1 is fa:16:3e:6c:f3:09,The mac address of vm2 is fa:16:3e:d2:14:b5, the action of the flow table is multiple ports. recirc_id(0x314123),in_port(8),ct_state(-new+est-rel-rpl),eth(src=fa:16:3e:d2:14:b5,dst=fa:16:3e:6c:f3:09),eth_type(0x0800),ipv4(proto=6,frag=no),tcp(dst=2048/0xf800), packets:60100, bytes:14257193, used:2.035s, flags:P., actions:push_vlan(vid=5,pcp=0),3,pop_vlan,push_vlan(vid=1732,pcp=0),1,pop_vlan,7,9,10,15,5,18 recirc_id(0x314123),in_port(8),ct_state(-new+est-rel+rpl-inv+trk),ct_zone(0x5),ct_mark(0),eth(src=fa:16:3e:d2:14:b5,dst=fa:16:3e:6c:f3:09),eth_type(0x0800),ipv4(frag=no), packets:1, bytes:54, used:2.037s, flags:., actions:push_vlan(vid=5,pcp=0),3,pop_vlan,push_vlan(vid=1732,pcp=0),1,pop_vlan,7,9,10,15,5,18 After careful debugging analysis, the openflow action in the flow table 82 added by neutron is the VM port instead of normal. As a result, the ovs br-int bridge cannot learn the mac entry of vm2. table=82, priority=77,ct_state=+new-est,tcp,reg5=0x9,tp_dst=0x4/0xfffc actions=ct(commit,zone=NXM_NX_REG6[0..15]),output:9 table=82, priority=77,ct_state=+est-rel-rpl,tcp,reg5=0x9,tp_dst=0x4/0xfffc actions=output:9 Looking forward to reply thanks. weichun ren

2 1

The outgoing unicast traffic from the VMS enabled in the security group is eventually forwarded to all ports.
by renwc＠chinatelecom.cn 31 May '24

31 May '24

Hello， Here is a description of a bug I encountered using openstack Security Group,If security groups are enabled for virtual nics on VMS, The outgoing traffic from the VMS enabled in the security group should be sent in unicast mode, but it is eventually forwarded to all ports in the vlan on the ovs br-int bridge. Does the openstack open source community have a patch to fix this bug? Can you provide patch fixed the issue? The following is my debugging process in detail. The unicast packet sent by the remote virtual machine 172.83.160.63 to the local virtual machine 172.83.160.74 is passed to the br-ex->br-int-> VM, Because the security group is enabled on the VM NIC, The br-int cannot learn the entry whose port is patch-ex and mac address is vm2. Therefore those packets sent from vm1 to vm2 are broadcast in vlans because they fail to match mac address entries. 172.83.160.74 patch-int 172.83.160.63 vm1 --- ovs(br-int) ------ ovs(br-ex)--- tor(physical switch) ......vm2 patch-ex The following is the ovs kernel flow table. The mac address of vm1 is fa:16:3e:6c:f3:09,The mac address of vm2 is fa:16:3e:d2:14:b5, the action of the flow table is multiple ports. recirc_id(0x314123),in_port(8),ct_state(-new+est-rel-rpl),eth(src=fa:16:3e:d2:14:b5,dst=fa:16:3e:6c:f3:09),eth_type(0x0800),ipv4(proto=6,frag=no),tcp(dst=2048/0xf800), packets:60100, bytes:14257193, used:2.035s, flags:P., actions:push_vlan(vid=5,pcp=0),3,pop_vlan,push_vlan(vid=1732,pcp=0),1,pop_vlan,7,9,10,15,5,18 recirc_id(0x314123),in_port(8),ct_state(-new+est-rel+rpl-inv+trk),ct_zone(0x5),ct_mark(0),eth(src=fa:16:3e:d2:14:b5,dst=fa:16:3e:6c:f3:09),eth_type(0x0800),ipv4(frag=no), packets:1, bytes:54, used:2.037s, flags:., actions:push_vlan(vid=5,pcp=0),3,pop_vlan,push_vlan(vid=1732,pcp=0),1,pop_vlan,7,9,10,15,5,18 After careful debugging analysis, the openflow action in the flow table 82 added by neutron is the VM port instead of normal. As a result, the ovs br-int bridge cannot learn the mac entry of vm2. table=82, priority=77,ct_state=+new-est,tcp,reg5=0x9,tp_dst=0x4/0xfffc actions=ct(commit,zone=NXM_NX_REG6[0..15]),output:9 table=82, priority=77,ct_state=+est-rel-rpl,tcp,reg5=0x9,tp_dst=0x4/0xfffc actions=output:9 Was the issue above fixed in the openstack ? Looking forward to reply thanks. Best Regards weichun ren

1 0

Re: [tc][all][senlin] Retiring the Senlin project
by Ghanshyam Mann 30 May '24

30 May '24

---- On Thu, 30 May 2024 11:43:00 -0700 Sang Tran Quoc wrote --- > > Dear Ghanshyam, > > Greeting! I’m Sang from Viet Nam and now I’m working at the FPT Smart Cloud (Silver member of OpenInfra) as the openstack team leader. We have teammate with about 10 members who maintains the senlin project on our production AutoScaling Feature (my presentation about senlin at last VietOpenInfra summithttps://www.youtube.com/watch?v=aYXDI4mwohQ&t=1s) It’s sad to hear about community retired this project from upstream so I would like to nominate our team to continue maintain this project and I will handle the PTL role for further roadmap, bugs fix and feature. > > Hopefully it’s not late, I’m look forward to hearing from you for further discuss about next chance for senlin project. ++ openstack-discuss ML Hi Sang, Thanks for reaching out and its good to hear about Senlin project usage and your interest in maintaining it. > Hopefully it’s not late, I’m look forward to hearing from you for further discuss about next chance for senlin project. Unfortunately, TC has already retired the Senlin project[1]. Since last cycle, we have been waiting for interested maintainers to step up. * No Leader in 2024.1 cycle ** https://etherpad.opendev.org/p/2024.1-leaderless * After TC discussion, Senlin has been marked as an 'inactive' project since '2024.1' cycle ** https://review.opendev.org/c/openstack/governance/+/902626 ** This was the time we were expecting someone to come forward and step up to maintain it. * No leader for Senlin in 2024.2 cycles also ** https://etherpad.opendev.org/p/2024.2-leaderless * Considering no leader for two cycles, no maintainer, and the process, TC initiated the retirement: ** https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack… But this is not the end and Senlin can still re-become the official OpenStack project[2]. For that, you need to re-apply Senlin as a new project in OpenStack, please check the process: - https://governance.openstack.org/tc/reference/new-projects-requirements.html [1] - https://review.opendev.org/c/openstack/governance/+/919347 - https://review.opendev.org/q/topic:%22retire-senlin%22+ [2]https://governance.openstack.org/tc/reference/dropping-projects.html#re-b… -gmann > > > Thanks & Best Regards > > > Sang Tran Quoc (Mr.) > FPT Smart Cloud > FPT Tower, 10 Pham Van Bach Str., Cau Giay Dist., Ha Noi, Viet Nam > E: sangtq8(a)fpt.com > T: 0962949843 > W: https://fptcloud.com | https://fpt.ai > > > > > From:Ghanshyam Mann gmann(a)ghanshyammann.com> > Date: Saturday, 11 May 2024 at 09:06 > To: openstack-discuss openstack-discuss(a)lists.openstack.org> > Subject: Re: [tc][all][senlin] Retiring the Senlin project > ---- On Wed, 24 Apr 2024 13:39:33 -0700 Ghanshyam Mann wrote --- > > Hello everyone, > > > > As you might know, the Senlin project is not active and was marked 'inactive' > > since 2024.1 cycle[1]. In this cycle also, this project is leaderless[2] and no maintainer > > stepped up to maintain it. As discussed in the Leaderless project etherpad[2], the next > > step is to retire the project. > > > > I'm sending this email to propose the Senlin retirement and based on the response, > > I will proceed next. > > Seeing no response to this email or no volunteer to maintain this project, I have proposed > the retirement in governance: > > - https://review.opendev.org/c/openstack/governance/+/919347 > > -gmann > > > > > [1] https://governance.openstack.org/tc/reference/emerging-technology-and-inact… > > [2] https://etherpad.opendev.org/p/2024.2-leaderless#L72 > > > > -gmann > > >

1 0