September 2022 - openstack-discuss

[release] Release countdown for week R-1, Sep 26 - 30
by Előd Illés 23 Sep '22

23 Sep '22

Development Focus ----------------- We are on the final mile of the Zed development cycle! Remember that the Zed final release will include the latest release candidate (for cycle-with-rc deliverables) or the latest intermediary release (for cycle-with-intermediary deliverables) available. September 29th, 2022 is the deadline for final Zed release candidates as well as any last cycle-with-intermediary deliverables. We will then enter a quiet period until we tag the final release on October 5th, 2022. Teams should be prioritizing fixing release-critical bugs, before that deadline. Otherwise it's time to start planning the 2023.1 Antelope development cycle, including discussing Forum and PTG sessions content, in preparation of Virtual PTG @ October 17-21, 2022. Actions ------- Watch for any translation patches coming through on the stable/zed branch and merge them quickly. If you discover a release-critical issue, please make sure to fix it on the master branch first, then backport the bugfix to the stable/zed branch before triggering a new release. Please drop by #openstack-release with any questions or concerns about the upcoming release ! Upcoming Deadlines & Dates -------------------------- Final Zed release: October 5th, 2022 Virtual PTG: October 17-21, 2022 Előd Illés irc: elodilles

1 0

[kolla] Antelope PTG
by Michał Nasiadka 23 Sep '22

23 Sep '22

Greetings Koalas, As you most probably know the OpenStack Antelope PTG is coming soon (17-21th Oct 2022). I’ve booked the usual slots (Mon-Wed) for Kolla project group (Kolla, Kolla-Ansible, Kayobe) with Wed being primary a slot for Kayobe. For operator-focus - the Kolla projects Operator Hour is booked for Tue, 16-17 UTC. Please sign up to the ether pad https://etherpad.opendev.org/p/kolla-antelope-ptg <https://etherpad.opendev.org/p/kolla-antelope-ptg> and propose topics. See you on the PTG! Best regards, Michal Nasiadka

1 0

[nova][ironic][ptg] Improved ironic<>nova driver interactions session
by Jay Faulkner 23 Sep '22

23 Sep '22

Hello Nova folks, At the Ironic PTG planning meeting this morning, discussion of improving Nova and Ironic driver interactions; and specifically this spec: https://review.opendev.org/c/openstack/nova-specs/+/842015/ came up. It's important for our teams to come together and reach consensus as to how to fix this. According to the latest OpenStack user survey ( https://www.openstack.org/analytics/) over 20% of Nova deployments utilize the Ironic driver. However, in my years operating Ironic for multiple different organizations, the Nova<>Ironic driver interface has repeatedly been the source of most operational pain. We should work together to prioritize saving these operators from downtime and bad failure semantics. I'd love for Ironic and Nova to schedule a session together to talk about this and try to find a solution. We owe it to our users to no longer shy away from this hard technical problem and fix things. What's the best way for us to get this time scheduled? Someone can feel free to reach out to me on IRC (JayF), or reply here and I'm sure we can work it out. Thanks in Advance, Jay Faulkner G-Research Open Source Software lab OpenStack Ironic PTL OpenStack Technical Committee member

2 1

Swift not recognizing mounted drives
by Andrew Boring 22 Sep '22

22 Sep '22

Hi all. I have a new Openstack Swift install, currently a single node with 12 disks running CentOS 8 Stream and Openstack Yoga / RDO release with Keystone Auth. Disks are formatted with xfs and mounted at /srv/node/sd[a-l] as described in the Install docs, but no Swift services reccognize it. SELinux is set to permissive mode. Log sample: Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sda is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdb is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdc is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdd is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sde is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdf is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdg is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdh is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdi is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdj is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdk is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdl is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Beginning replication run Sep 22 11:29:01 [hostname] account-replicator[182475]: Replication run OVER Sep 22 11:29:01 [hostname] account-replicator[182475]: Attempted to replicate 0 dbs in 0.00344 seconds (0.00000/s) Sep 22 11:29:01 [hostname] account-replicator[182475]: Removed 0 dbs Sep 22 11:29:01 [hostname] account-replicator[182475]: 0 successes, 144 failures Sep 22 11:29:01 [hostname] account-replicator[182475]: diff:0 diff_capped:0 empty:0 hashmatch:0 no_change:0 remote_merge:0 rsync:0 ts_repl:0 Container and Object processes show the same issue. And yet...devices are mounted: [root@swift]# mount | grep srv /dev/sdb on /srv/node/sdb type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdc on /srv/node/sdc type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdd on /srv/node/sdd type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sde on /srv/node/sde type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdf on /srv/node/sdf type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdg on /srv/node/sdg type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdh on /srv/node/sdh type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdi on /srv/node/sdi type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdj on /srv/node/sdj type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdk on /srv/node/sdk type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdl on /srv/node/sdl type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sda on /srv/node/sda type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) And the rings are built: [root@swift]# swift-ring-builder /etc/swift/account.builder /etc/swift/account.builder, build version 13, id 415987b4753847e9924147119872d547 2048 partitions, 3.000000 replicas, 1 regions, 1 zones, 12 devices, 0.00 balance, 0.00 dispersion The minimum number of hours before a partition can be reassigned is 1 (0:16:41 remaining) The overload factor is 0.00% (0.000000) Ring file /etc/swift/account.ring.gz is up-to-date Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta 0 1 1 172.16.2.2:6202 172.16.2.2:6202 sda 100.00 512 0.00 1 1 1 172.16.2.2:6202 172.16.2.2:6202 sdb 100.00 512 0.00 2 1 1 172.16.2.2:6202 172.16.2.2:6202 sdc 100.00 512 0.00 3 1 1 172.16.2.2:6202 172.16.2.2:6202 sdd 100.00 512 0.00 4 1 1 172.16.2.2:6202 172.16.2.2:6202 sde 100.00 512 0.00 5 1 1 172.16.2.2:6202 172.16.2.2:6202 sdf 100.00 512 0.00 6 1 1 172.16.2.2:6202 172.16.2.2:6202 sdg 100.00 512 0.00 7 1 1 172.16.2.2:6202 172.16.2.2:6202 sdh 100.00 512 0.00 8 1 1 172.16.2.2:6202 172.16.2.2:6202 sdi 100.00 512 0.00 9 1 1 172.16.2.2:6202 172.16.2.2:6202 sdj 100.00 512 0.00 10 1 1 172.16.2.2:6202 172.16.2.2:6202 sdk 100.00 512 0.00 11 1 1 172.16.2.2:6202 172.16.2.2:6202 sdl 100.00 512 0.00 Bash certainly thinks they are actually mounted and writable: [root@swift ~]# ls -l /srv/node/sda total 0 [root@swift ~]# touch /srv/node/sda/file [root@swift ~]# ls /srv/node/sda file [root@swift ~]# umount /dev/sda [root@swift ~]# ls /srv/node/sda [root@swift ~]# mount /srv/node/sda/ [root@swift ~]# ls /srv/node/sda file [root@swift ~]# rm /srv/node/sda/file rm: remove regular empty file '/srv/node/sda/file'? y And Swift is configured to look at /srv/node for the devices: [root@swift ~]# grep devices /etc/swift/*.conf /etc/swift/account-server.conf:devices = /srv/node /etc/swift/container-server.conf:devices = /srv/node /etc/swift/object-server.conf:devices = /srv/node Running "swift info" shows the expected configuration output, but "swift stat" throws this error: [root@controller ~]# swift stat Account HEAD failed: https:/[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8 403 Forbidden Failed Transaction ID: tx9128250987cb4e07ad55d-00632c8d05 and the logs show account-server throwing "507 ERROR Insufficient Storage" before 503, and then returning a 403. [root@swift ~]# journalctl -b | grep tx9128250987cb4e07ad55d-00632c8d05 Sep 22 12:27:49 [hostname] account-server[182505]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdc/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0004 "-" 182505 - Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdc (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:49 [hostname] account-server[182515]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdj/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0004 "-" 182515 - Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdj (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:49 [hostname] account-server[182514]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdh/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182514 - Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdh (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:49 [hostname] account-server[182506]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdl/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182506 - Sep 22 12:27:49 [hostname] account-server[182504]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdf/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0010 "-" 182504 - Sep 22 12:27:49 [hostname] account-server[182504]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdi/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0002 "-" 182504 - Sep 22 12:27:49 [hostname] proxy-server[182464]: Handoff requested (4) (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] account-server[182512]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sda/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182512 - Sep 22 12:27:50 [hostname] proxy-server[182464]: Handoff requested (5) (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] account-server[182494]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sde/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0010 "-" 182494 - Sep 22 12:27:50 [hostname] proxy-server[182464]: Handoff requested (6) (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] account-server[182506]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sdk/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0003 "-" 182506 - Sep 22 12:27:50 [hostname] proxy-server[182464]: Account HEAD returning 503 for [507, 507, 507] (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] proxy-server[182464]: - - 22/Sep/2022/16/27/50 HEAD /v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8%3Fformat%3Djson HTTP/1.0 503 - Swift - - - - tx9128250987cb4e07ad55d-00632c8d05 - 0.0337 RL - 1663864069.976659298 1663864070.010398626 - Sep 22 12:27:51 [hostname] proxy-server[182464]: [IP] 172.16.2.2 22/Sep/2022/16/27/51 HEAD /v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8%3Fformat%3Djson HTTP/1.0 403 - python-swiftclient-3.13.1 gAAAAABjLI0FH9MP... - - - tx9128250987cb4e07ad55d-00632c8d05 - 1.4298 - - 1663864069.970289707 1663864071.400121450 - No data is being written to the disks by Swift: [root@swift ~]# ls -lR /srv/node /srv/node: total 0 drwxr-xr-x. 2 swift swift 6 Sep 22 11:13 sda drwxr-xr-x. 2 swift swift 6 Sep 22 11:07 sdb drwxr-xr-x. 2 swift swift 6 Sep 22 11:07 sdc ... /srv/node/sda: total 0 /srv/node/sdb: total 0 /srv/node/sdc: total 0 ... What am I missing here?

2 5

[dev][security-sig] Revisiting tarfile, or "What's old is new again"
by Jeremy Stanley 22 Sep '22

22 Sep '22

The tarfile module from Python's standard library is in the news this week, with people publicly exploiting the very long-standing CVE-2007-4559[0] (yes, you read that correctly, *2007*). Some old-timers in the community might remember this from such popular hits as OSSA-2011-001: Path traversal issues registering malicious images using EC2 API[1], our very first OpenStack Security Advisory! This revived interest in unsafe use of tarfile methods will undoubtedly have lots of people scanning OpenStack's Git repos looking for potentially exploitable calls. Indeed, some of our own community members are already auditing the collective codebase to make sure new vulnerabilities haven't sneaked in over the 11 years since this first came up for us, but more help is always welcome. I encourage anyone using tarfile in their projects to double-check you're doing so safely[2]. If you rely bandit to check your source code, be advised that the most recent 1.7.4 release doesn't catch this but you can install its main branch[3] instead which does include a check for it, at least until they tag a new release (which I have a feeling they'll do quite soon given the recent furor around this topic). On a related note, I want to take this opportunity to remind everyone that OpenStack has a Security Special Interest Group (SIG), which meets monthly[4] on IRC, and members will also be in attendance at the upcoming virtual PTG[5] in case anyone is interested in discussing this or similar subject matter. Our PTG slot is currently booked for 15:00 UTC Wednesday (2022-10-19), though we can adjust or book an additional hour at another time if this conflicts with any tracks people also need to join, just let me know. [0] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4559 [1] https://security.openstack.org/ossa/OSSA-2011-001.html [2] https://docs.python.org/3/library/tarfile.html#tarfile.TarFile.extractall [3] https://github.com/pycqa/bandit [4] https://meetings.opendev.org/#OpenStack_Security_SIG_meeting [5] https://ptg.opendev.org/ptg.html -- Jeremy Stanley

2 1

[ironic][ptg] Last call for topics
by Jay Faulkner 22 Sep '22

22 Sep '22

This morning, a group of Ironic cores met and mapped out a rough schedule for our PTG sessions. Please review the etherpad here: https://etherpad.opendev.org/p/ironic-antelope-ptg for more details on the outcome of that. We did want to give one last chance for anyone who has not yet had a chance to add their topic to do so before the plans are finalized. Please add any additional topics to the etherpad, and preferably, also respond to this email or let us know in #openstack-ironic on IRC. Thanks, Jay Faulkner G-Research Open Source Software lab OpenStack Ironic PTL OpenStack Technical Committee member

1 0

How do I disable replication in swift
by Paladox 22 Sep '22

22 Sep '22

Hello, I would like Swift to only have 1 file not any copies of it, how do I disable replication as you must set the value to 1 when you create the rings? (We don’t have the disk space to have replication / copies of the files)

2 2

[neutron] Drivers meeting - Friday 22.09.2022 - cancelled
by Lajos Katona 22 Sep '22

22 Sep '22

Hi Neutron Drivers! Due to the lack of agenda, let's cancel tomorrow's drivers meeting. See You on the meeting next week. Lajos Katona (lajoskatona)

1 0

Cinder-volume active active setup
by Eugen Block 22 Sep '22

22 Sep '22

Hi, I didn't mean to hijack the other thread so I'll start a new one. There are some pages I found incl. Gorkas article [1], but I don't really understand yet how to configure it. We don't use any of the automated deployments (we created our own) like TripleO etc., is there any guide showing how to setup cinder-volume active/active? I see in my lab environment that python3-tooz is already installed on the control node, but how do I use it? Besides the "cluster" config option in the cinder.conf (is that defined when setting up the DLM?) what else is required? I also found this thread [2] pointing to the source code, but that doesn't really help me at this point. Any pointers to a how-to or deployment guide would be highly appreciated! Thanks, Eugen [1] https://gorka.eguileor.com/a-cinder-road-to-activeactive-ha/ [2] https://www.mail-archive.com/openstack@lists.openstack.org/msg18385.html

2 5

Floating IP attached to octavia not working | tripleo wallaby | centos | OVN DVR
by Swogat Pradhan 22 Sep '22

22 Sep '22

Hi, I've deployed Octavia + OVN + DVR and floating IPs assigned to Octavia VIPs are not reachable. Other floating IPs tied to nova instances are working fine. The octavia vip is reachable and working fine from inside on the private network, however the Floating IP is not reachable at all. I found a similar issue in bugzilla but the bug is closed now and i do not have any solution. https://bugzilla.redhat.com/show_bug.cgi?id=1608951 Please help me get a solution for this issue. With regards, Swogat pradhan

3 3