September 2022 - openstack-discuss

[nova][ironic][ptg] Improved ironic<>nova driver interactions session
by Jay Faulkner 23 Sep '22

23 Sep '22

Hello Nova folks, At the Ironic PTG planning meeting this morning, discussion of improving Nova and Ironic driver interactions; and specifically this spec: https://review.opendev.org/c/openstack/nova-specs/+/842015/ came up. It's important for our teams to come together and reach consensus as to how to fix this. According to the latest OpenStack user survey ( https://www.openstack.org/analytics/) over 20% of Nova deployments utilize the Ironic driver. However, in my years operating Ironic for multiple different organizations, the Nova<>Ironic driver interface has repeatedly been the source of most operational pain. We should work together to prioritize saving these operators from downtime and bad failure semantics. I'd love for Ironic and Nova to schedule a session together to talk about this and try to find a solution. We owe it to our users to no longer shy away from this hard technical problem and fix things. What's the best way for us to get this time scheduled? Someone can feel free to reach out to me on IRC (JayF), or reply here and I'm sure we can work it out. Thanks in Advance, Jay Faulkner G-Research Open Source Software lab OpenStack Ironic PTL OpenStack Technical Committee member

2 1

Swift not recognizing mounted drives
by Andrew Boring 23 Sep '22

23 Sep '22

Hi all. I have a new Openstack Swift install, currently a single node with 12 disks running CentOS 8 Stream and Openstack Yoga / RDO release with Keystone Auth. Disks are formatted with xfs and mounted at /srv/node/sd[a-l] as described in the Install docs, but no Swift services reccognize it. SELinux is set to permissive mode. Log sample: Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sda is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdb is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdc is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdd is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sde is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdf is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdg is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdh is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdi is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdj is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdk is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdl is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Beginning replication run Sep 22 11:29:01 [hostname] account-replicator[182475]: Replication run OVER Sep 22 11:29:01 [hostname] account-replicator[182475]: Attempted to replicate 0 dbs in 0.00344 seconds (0.00000/s) Sep 22 11:29:01 [hostname] account-replicator[182475]: Removed 0 dbs Sep 22 11:29:01 [hostname] account-replicator[182475]: 0 successes, 144 failures Sep 22 11:29:01 [hostname] account-replicator[182475]: diff:0 diff_capped:0 empty:0 hashmatch:0 no_change:0 remote_merge:0 rsync:0 ts_repl:0 Container and Object processes show the same issue. And yet...devices are mounted: [root@swift]# mount | grep srv /dev/sdb on /srv/node/sdb type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdc on /srv/node/sdc type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdd on /srv/node/sdd type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sde on /srv/node/sde type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdf on /srv/node/sdf type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdg on /srv/node/sdg type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdh on /srv/node/sdh type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdi on /srv/node/sdi type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdj on /srv/node/sdj type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdk on /srv/node/sdk type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdl on /srv/node/sdl type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sda on /srv/node/sda type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) And the rings are built: [root@swift]# swift-ring-builder /etc/swift/account.builder /etc/swift/account.builder, build version 13, id 415987b4753847e9924147119872d547 2048 partitions, 3.000000 replicas, 1 regions, 1 zones, 12 devices, 0.00 balance, 0.00 dispersion The minimum number of hours before a partition can be reassigned is 1 (0:16:41 remaining) The overload factor is 0.00% (0.000000) Ring file /etc/swift/account.ring.gz is up-to-date Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta 0 1 1 172.16.2.2:6202 172.16.2.2:6202 sda 100.00 512 0.00 1 1 1 172.16.2.2:6202 172.16.2.2:6202 sdb 100.00 512 0.00 2 1 1 172.16.2.2:6202 172.16.2.2:6202 sdc 100.00 512 0.00 3 1 1 172.16.2.2:6202 172.16.2.2:6202 sdd 100.00 512 0.00 4 1 1 172.16.2.2:6202 172.16.2.2:6202 sde 100.00 512 0.00 5 1 1 172.16.2.2:6202 172.16.2.2:6202 sdf 100.00 512 0.00 6 1 1 172.16.2.2:6202 172.16.2.2:6202 sdg 100.00 512 0.00 7 1 1 172.16.2.2:6202 172.16.2.2:6202 sdh 100.00 512 0.00 8 1 1 172.16.2.2:6202 172.16.2.2:6202 sdi 100.00 512 0.00 9 1 1 172.16.2.2:6202 172.16.2.2:6202 sdj 100.00 512 0.00 10 1 1 172.16.2.2:6202 172.16.2.2:6202 sdk 100.00 512 0.00 11 1 1 172.16.2.2:6202 172.16.2.2:6202 sdl 100.00 512 0.00 Bash certainly thinks they are actually mounted and writable: [root@swift ~]# ls -l /srv/node/sda total 0 [root@swift ~]# touch /srv/node/sda/file [root@swift ~]# ls /srv/node/sda file [root@swift ~]# umount /dev/sda [root@swift ~]# ls /srv/node/sda [root@swift ~]# mount /srv/node/sda/ [root@swift ~]# ls /srv/node/sda file [root@swift ~]# rm /srv/node/sda/file rm: remove regular empty file '/srv/node/sda/file'? y And Swift is configured to look at /srv/node for the devices: [root@swift ~]# grep devices /etc/swift/*.conf /etc/swift/account-server.conf:devices = /srv/node /etc/swift/container-server.conf:devices = /srv/node /etc/swift/object-server.conf:devices = /srv/node Running "swift info" shows the expected configuration output, but "swift stat" throws this error: [root@controller ~]# swift stat Account HEAD failed: https:/[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8 403 Forbidden Failed Transaction ID: tx9128250987cb4e07ad55d-00632c8d05 and the logs show account-server throwing "507 ERROR Insufficient Storage" before 503, and then returning a 403. [root@swift ~]# journalctl -b | grep tx9128250987cb4e07ad55d-00632c8d05 Sep 22 12:27:49 [hostname] account-server[182505]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdc/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0004 "-" 182505 - Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdc (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:49 [hostname] account-server[182515]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdj/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0004 "-" 182515 - Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdj (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:49 [hostname] account-server[182514]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdh/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182514 - Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdh (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:49 [hostname] account-server[182506]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdl/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182506 - Sep 22 12:27:49 [hostname] account-server[182504]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdf/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0010 "-" 182504 - Sep 22 12:27:49 [hostname] account-server[182504]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdi/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0002 "-" 182504 - Sep 22 12:27:49 [hostname] proxy-server[182464]: Handoff requested (4) (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] account-server[182512]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sda/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182512 - Sep 22 12:27:50 [hostname] proxy-server[182464]: Handoff requested (5) (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] account-server[182494]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sde/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0010 "-" 182494 - Sep 22 12:27:50 [hostname] proxy-server[182464]: Handoff requested (6) (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] account-server[182506]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sdk/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0003 "-" 182506 - Sep 22 12:27:50 [hostname] proxy-server[182464]: Account HEAD returning 503 for [507, 507, 507] (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] proxy-server[182464]: - - 22/Sep/2022/16/27/50 HEAD /v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8%3Fformat%3Djson HTTP/1.0 503 - Swift - - - - tx9128250987cb4e07ad55d-00632c8d05 - 0.0337 RL - 1663864069.976659298 1663864070.010398626 - Sep 22 12:27:51 [hostname] proxy-server[182464]: [IP] 172.16.2.2 22/Sep/2022/16/27/51 HEAD /v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8%3Fformat%3Djson HTTP/1.0 403 - python-swiftclient-3.13.1 gAAAAABjLI0FH9MP... - - - tx9128250987cb4e07ad55d-00632c8d05 - 1.4298 - - 1663864069.970289707 1663864071.400121450 - No data is being written to the disks by Swift: [root@swift ~]# ls -lR /srv/node /srv/node: total 0 drwxr-xr-x. 2 swift swift 6 Sep 22 11:13 sda drwxr-xr-x. 2 swift swift 6 Sep 22 11:07 sdb drwxr-xr-x. 2 swift swift 6 Sep 22 11:07 sdc ... /srv/node/sda: total 0 /srv/node/sdb: total 0 /srv/node/sdc: total 0 ... What am I missing here?

2 5

[dev][security-sig] Revisiting tarfile, or "What's old is new again"
by Jeremy Stanley 23 Sep '22

23 Sep '22

The tarfile module from Python's standard library is in the news this week, with people publicly exploiting the very long-standing CVE-2007-4559[0] (yes, you read that correctly, *2007*). Some old-timers in the community might remember this from such popular hits as OSSA-2011-001: Path traversal issues registering malicious images using EC2 API[1], our very first OpenStack Security Advisory! This revived interest in unsafe use of tarfile methods will undoubtedly have lots of people scanning OpenStack's Git repos looking for potentially exploitable calls. Indeed, some of our own community members are already auditing the collective codebase to make sure new vulnerabilities haven't sneaked in over the 11 years since this first came up for us, but more help is always welcome. I encourage anyone using tarfile in their projects to double-check you're doing so safely[2]. If you rely bandit to check your source code, be advised that the most recent 1.7.4 release doesn't catch this but you can install its main branch[3] instead which does include a check for it, at least until they tag a new release (which I have a feeling they'll do quite soon given the recent furor around this topic). On a related note, I want to take this opportunity to remind everyone that OpenStack has a Security Special Interest Group (SIG), which meets monthly[4] on IRC, and members will also be in attendance at the upcoming virtual PTG[5] in case anyone is interested in discussing this or similar subject matter. Our PTG slot is currently booked for 15:00 UTC Wednesday (2022-10-19), though we can adjust or book an additional hour at another time if this conflicts with any tracks people also need to join, just let me know. [0] https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2007-4559 [1] https://security.openstack.org/ossa/OSSA-2011-001.html [2] https://docs.python.org/3/library/tarfile.html#tarfile.TarFile.extractall [3] https://github.com/pycqa/bandit [4] https://meetings.opendev.org/#OpenStack_Security_SIG_meeting [5] https://ptg.opendev.org/ptg.html -- Jeremy Stanley

2 1

[ironic][ptg] Last call for topics
by Jay Faulkner 23 Sep '22

23 Sep '22

This morning, a group of Ironic cores met and mapped out a rough schedule for our PTG sessions. Please review the etherpad here: https://etherpad.opendev.org/p/ironic-antelope-ptg for more details on the outcome of that. We did want to give one last chance for anyone who has not yet had a chance to add their topic to do so before the plans are finalized. Please add any additional topics to the etherpad, and preferably, also respond to this email or let us know in #openstack-ironic on IRC. Thanks, Jay Faulkner G-Research Open Source Software lab OpenStack Ironic PTL OpenStack Technical Committee member

1 0

How do I disable replication in swift
by Paladox 23 Sep '22

23 Sep '22

Hello, I would like Swift to only have 1 file not any copies of it, how do I disable replication as you must set the value to 1 when you create the rings? (We don’t have the disk space to have replication / copies of the files)

2 2

[neutron] Drivers meeting - Friday 22.09.2022 - cancelled
by Lajos Katona 23 Sep '22

23 Sep '22

Hi Neutron Drivers! Due to the lack of agenda, let's cancel tomorrow's drivers meeting. See You on the meeting next week. Lajos Katona (lajoskatona)

1 0

Cinder-volume active active setup
by Eugen Block 22 Sep '22

22 Sep '22

Hi, I didn't mean to hijack the other thread so I'll start a new one. There are some pages I found incl. Gorkas article [1], but I don't really understand yet how to configure it. We don't use any of the automated deployments (we created our own) like TripleO etc., is there any guide showing how to setup cinder-volume active/active? I see in my lab environment that python3-tooz is already installed on the control node, but how do I use it? Besides the "cluster" config option in the cinder.conf (is that defined when setting up the DLM?) what else is required? I also found this thread [2] pointing to the source code, but that doesn't really help me at this point. Any pointers to a how-to or deployment guide would be highly appreciated! Thanks, Eugen [1] https://gorka.eguileor.com/a-cinder-road-to-activeactive-ha/ [2] https://www.mail-archive.com/openstack@lists.openstack.org/msg18385.html

2 5

Floating IP attached to octavia not working | tripleo wallaby | centos | OVN DVR
by Swogat Pradhan 22 Sep '22

22 Sep '22

Hi, I've deployed Octavia + OVN + DVR and floating IPs assigned to Octavia VIPs are not reachable. Other floating IPs tied to nova instances are working fine. The octavia vip is reachable and working fine from inside on the private network, however the Floating IP is not reachable at all. I found a similar issue in bugzilla but the bug is closed now and i do not have any solution. https://bugzilla.redhat.com/show_bug.cgi?id=1608951 Please help me get a solution for this issue. With regards, Swogat pradhan

3 3

[ansible-os-modules] Antelope PTG in October | Ansible OpenStack Collection
by Jakob Meng 22 Sep '22

22 Sep '22

Hi everyone, we have scheduled our Antelope's vPTG session for the Ansible OpenStack Collection [1] at the same time and place as in April: at 13:00 - 15:00 UTC on Thursday October 20, 2022 in Essex room. Topics will be our migration progress to openstacksdk's first major release, new features, notable changes and open todos. Do not hesitate to add any questions and topics you want to address in our PTG session to etherpad [2]. Looking forward to meet you ☺️ [1] https://opendev.org/openstack/ansible-collections-openstack [2] https://etherpad.opendev.org/p/oct2022-ptg-os-ansible-modules Thanks, Ananya, Arx, Rafael, Sagi and Jakob

1 0

[openstack][quuens][cinder] consistency groups bug ?
by Ignazio Cassano 22 Sep '22

22 Sep '22

Hi everyone, in my openstack installation, for performance reasons, I use a cinder type to address different storage virtual machines on the same device. Balancing takes place via the scheduler. Below is an example of my configuration in cinder.conf: https://paste.openstack.org/show/b0hCTsdniLhinoQiVT7A/ As you can see nfsgold1 to nfsgold8 are all part of the volume type nfsgold. Unfortunately if I create a consistency group with volume type nfsgold, it can only contain volumes on the same backend, even though they are of the same type. For example, if in my consistency group there is a volume on nfsgold1 and I add a volume present on nfsgold8 I get the following: https://paste.openstack.org/show/b4XnUeGDF5wYlCBLTcuc/ Any help, please ? Ignazio

2 3