Hi all.
I have a new Openstack Swift install, currently a single node with 12 disks running CentOS 8 Stream and Openstack Yoga / RDO release with Keystone Auth. Disks are formatted with xfs and mounted at /srv/node/sd[a-l] as described in the Install docs, but no Swift services reccognize it.
SELinux is set to permissive mode.
Log sample: Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sda is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdb is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdc is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdd is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sde is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdf is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdg is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdh is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdi is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdj is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdk is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdl is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Beginning replication run Sep 22 11:29:01 [hostname] account-replicator[182475]: Replication run OVER Sep 22 11:29:01 [hostname] account-replicator[182475]: Attempted to replicate 0 dbs in 0.00344 seconds (0.00000/s) Sep 22 11:29:01 [hostname] account-replicator[182475]: Removed 0 dbs Sep 22 11:29:01 [hostname] account-replicator[182475]: 0 successes, 144 failures Sep 22 11:29:01 [hostname] account-replicator[182475]: diff:0 diff_capped:0 empty:0 hashmatch:0 no_change:0 remote_merge:0 rsync:0 ts_repl:0
Container and Object processes show the same issue. And yet...devices are mounted:
[root@swift]# mount | grep srv /dev/sdb on /srv/node/sdb type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdc on /srv/node/sdc type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdd on /srv/node/sdd type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sde on /srv/node/sde type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdf on /srv/node/sdf type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdg on /srv/node/sdg type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdh on /srv/node/sdh type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdi on /srv/node/sdi type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdj on /srv/node/sdj type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdk on /srv/node/sdk type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdl on /srv/node/sdl type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sda on /srv/node/sda type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
And the rings are built:
[root@swift]# swift-ring-builder /etc/swift/account.builder /etc/swift/account.builder, build version 13, id 415987b4753847e9924147119872d547 2048 partitions, 3.000000 replicas, 1 regions, 1 zones, 12 devices, 0.00 balance, 0.00 dispersion The minimum number of hours before a partition can be reassigned is 1 (0:16:41 remaining) The overload factor is 0.00% (0.000000) Ring file /etc/swift/account.ring.gz is up-to-date Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta 0 1 1 172.16.2.2:6202 172.16.2.2:6202 sda 100.00 512 0.00 1 1 1 172.16.2.2:6202 172.16.2.2:6202 sdb 100.00 512 0.00 2 1 1 172.16.2.2:6202 172.16.2.2:6202 sdc 100.00 512 0.00 3 1 1 172.16.2.2:6202 172.16.2.2:6202 sdd 100.00 512 0.00 4 1 1 172.16.2.2:6202 172.16.2.2:6202 sde 100.00 512 0.00 5 1 1 172.16.2.2:6202 172.16.2.2:6202 sdf 100.00 512 0.00 6 1 1 172.16.2.2:6202 172.16.2.2:6202 sdg 100.00 512 0.00 7 1 1 172.16.2.2:6202 172.16.2.2:6202 sdh 100.00 512 0.00 8 1 1 172.16.2.2:6202 172.16.2.2:6202 sdi 100.00 512 0.00 9 1 1 172.16.2.2:6202 172.16.2.2:6202 sdj 100.00 512 0.00 10 1 1 172.16.2.2:6202 172.16.2.2:6202 sdk 100.00 512 0.00 11 1 1 172.16.2.2:6202 172.16.2.2:6202 sdl 100.00 512 0.00
Bash certainly thinks they are actually mounted and writable:
[root@swift ~]# ls -l /srv/node/sda total 0 [root@swift ~]# touch /srv/node/sda/file [root@swift ~]# ls /srv/node/sda file [root@swift ~]# umount /dev/sda [root@swift ~]# ls /srv/node/sda [root@swift ~]# mount /srv/node/sda/ [root@swift ~]# ls /srv/node/sda file [root@swift ~]# rm /srv/node/sda/file rm: remove regular empty file '/srv/node/sda/file'? y
And Swift is configured to look at /srv/node for the devices:
[root@swift ~]# grep devices /etc/swift/*.conf /etc/swift/account-server.conf:devices = /srv/node /etc/swift/container-server.conf:devices = /srv/node /etc/swift/object-server.conf:devices = /srv/node
Running "swift info" shows the expected configuration output, but "swift stat" throws this error:
[root@controller ~]# swift stat Account HEAD failed: https:/[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8 403 Forbidden Failed Transaction ID: tx9128250987cb4e07ad55d-00632c8d05
and the logs show account-server throwing "507 ERROR Insufficient Storage" before 503, and then returning a 403.
[root@swift ~]# journalctl -b | grep tx9128250987cb4e07ad55d-00632c8d05 Sep 22 12:27:49 [hostname] account-server[182505]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdc/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://%5Bapi endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0004 "-" 182505 - Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdc (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:49 [hostname] account-server[182515]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdj/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://%5Bapi endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0004 "-" 182515 - Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdj (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:49 [hostname] account-server[182514]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdh/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://%5Bapi endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182514 - Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdh (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:49 [hostname] account-server[182506]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdl/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://%5Bapi endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182506 - Sep 22 12:27:49 [hostname] account-server[182504]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdf/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://%5Bapi endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0010 "-" 182504 - Sep 22 12:27:49 [hostname] account-server[182504]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdi/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://%5Bapi endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0002 "-" 182504 - Sep 22 12:27:49 [hostname] proxy-server[182464]: Handoff requested (4) (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] account-server[182512]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sda/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://%5Bapi endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182512 - Sep 22 12:27:50 [hostname] proxy-server[182464]: Handoff requested (5) (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] account-server[182494]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sde/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://%5Bapi endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0010 "-" 182494 - Sep 22 12:27:50 [hostname] proxy-server[182464]: Handoff requested (6) (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] account-server[182506]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sdk/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://%5Bapi endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0003 "-" 182506 - Sep 22 12:27:50 [hostname] proxy-server[182464]: Account HEAD returning 503 for [507, 507, 507] (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] proxy-server[182464]: - - 22/Sep/2022/16/27/50 HEAD /v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8%3Fformat%3Djson HTTP/1.0 503 - Swift - - - - tx9128250987cb4e07ad55d-00632c8d05 - 0.0337 RL - 1663864069.976659298 1663864070.010398626 - Sep 22 12:27:51 [hostname] proxy-server[182464]: [IP] 172.16.2.2 22/Sep/2022/16/27/51 HEAD /v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8%3Fformat%3Djson HTTP/1.0 403 - python-swiftclient-3.13.1 gAAAAABjLI0FH9MP... - - - tx9128250987cb4e07ad55d-00632c8d05 - 1.4298 - - 1663864069.970289707 1663864071.400121450 -
No data is being written to the disks by Swift:
[root@swift ~]# ls -lR /srv/node /srv/node: total 0 drwxr-xr-x. 2 swift swift 6 Sep 22 11:13 sda drwxr-xr-x. 2 swift swift 6 Sep 22 11:07 sdb drwxr-xr-x. 2 swift swift 6 Sep 22 11:07 sdc ...
/srv/node/sda: total 0
/srv/node/sdb: total 0
/srv/node/sdc: total 0 ...
What am I missing here?