Swift not recognizing mounted drives
Andrew Boring
andrew at andrewboring.com
Thu Sep 22 16:57:25 UTC 2022
Hi all.
I have a new Openstack Swift install, currently a single node with 12 disks running CentOS 8 Stream and Openstack Yoga / RDO release with Keystone Auth.
Disks are formatted with xfs and mounted at /srv/node/sd[a-l] as described in the Install docs, but no Swift services reccognize it.
SELinux is set to permissive mode.
Log sample:
Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sda is not mounted
Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdb is not mounted
Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdc is not mounted
Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdd is not mounted
Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sde is not mounted
Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdf is not mounted
Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdg is not mounted
Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdh is not mounted
Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdi is not mounted
Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdj is not mounted
Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdk is not mounted
Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdl is not mounted
Sep 22 11:29:01 [hostname] account-replicator[182475]: Beginning replication run
Sep 22 11:29:01 [hostname] account-replicator[182475]: Replication run OVER
Sep 22 11:29:01 [hostname] account-replicator[182475]: Attempted to replicate 0 dbs in 0.00344 seconds (0.00000/s)
Sep 22 11:29:01 [hostname] account-replicator[182475]: Removed 0 dbs
Sep 22 11:29:01 [hostname] account-replicator[182475]: 0 successes, 144 failures
Sep 22 11:29:01 [hostname] account-replicator[182475]: diff:0 diff_capped:0 empty:0 hashmatch:0 no_change:0 remote_merge:0 rsync:0 ts_repl:0
Container and Object processes show the same issue.
And yet...devices are mounted:
[root at swift]# mount | grep srv
/dev/sdb on /srv/node/sdb type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdc on /srv/node/sdc type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdd on /srv/node/sdd type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sde on /srv/node/sde type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdf on /srv/node/sdf type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdg on /srv/node/sdg type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdh on /srv/node/sdh type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdi on /srv/node/sdi type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdj on /srv/node/sdj type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdk on /srv/node/sdk type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sdl on /srv/node/sdl type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
/dev/sda on /srv/node/sda type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota)
And the rings are built:
[root at swift]# swift-ring-builder /etc/swift/account.builder
/etc/swift/account.builder, build version 13, id 415987b4753847e9924147119872d547
2048 partitions, 3.000000 replicas, 1 regions, 1 zones, 12 devices, 0.00 balance, 0.00 dispersion
The minimum number of hours before a partition can be reassigned is 1 (0:16:41 remaining)
The overload factor is 0.00% (0.000000)
Ring file /etc/swift/account.ring.gz is up-to-date
Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta
0 1 1 172.16.2.2:6202 172.16.2.2:6202 sda 100.00 512 0.00
1 1 1 172.16.2.2:6202 172.16.2.2:6202 sdb 100.00 512 0.00
2 1 1 172.16.2.2:6202 172.16.2.2:6202 sdc 100.00 512 0.00
3 1 1 172.16.2.2:6202 172.16.2.2:6202 sdd 100.00 512 0.00
4 1 1 172.16.2.2:6202 172.16.2.2:6202 sde 100.00 512 0.00
5 1 1 172.16.2.2:6202 172.16.2.2:6202 sdf 100.00 512 0.00
6 1 1 172.16.2.2:6202 172.16.2.2:6202 sdg 100.00 512 0.00
7 1 1 172.16.2.2:6202 172.16.2.2:6202 sdh 100.00 512 0.00
8 1 1 172.16.2.2:6202 172.16.2.2:6202 sdi 100.00 512 0.00
9 1 1 172.16.2.2:6202 172.16.2.2:6202 sdj 100.00 512 0.00
10 1 1 172.16.2.2:6202 172.16.2.2:6202 sdk 100.00 512 0.00
11 1 1 172.16.2.2:6202 172.16.2.2:6202 sdl 100.00 512 0.00
Bash certainly thinks they are actually mounted and writable:
[root at swift ~]# ls -l /srv/node/sda
total 0
[root at swift ~]# touch /srv/node/sda/file
[root at swift ~]# ls /srv/node/sda
file
[root at swift ~]# umount /dev/sda
[root at swift ~]# ls /srv/node/sda
[root at swift ~]# mount /srv/node/sda/
[root at swift ~]# ls /srv/node/sda
file
[root at swift ~]# rm /srv/node/sda/file
rm: remove regular empty file '/srv/node/sda/file'? y
And Swift is configured to look at /srv/node for the devices:
[root at swift ~]# grep devices /etc/swift/*.conf
/etc/swift/account-server.conf:devices = /srv/node
/etc/swift/container-server.conf:devices = /srv/node
/etc/swift/object-server.conf:devices = /srv/node
Running "swift info" shows the expected configuration output, but "swift stat" throws this error:
[root at controller ~]# swift stat
Account HEAD failed: https:/[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8 403 Forbidden
Failed Transaction ID: tx9128250987cb4e07ad55d-00632c8d05
and the logs show account-server throwing "507 ERROR Insufficient Storage" before 503, and then returning a 403.
[root at swift ~]# journalctl -b | grep tx9128250987cb4e07ad55d-00632c8d05
Sep 22 12:27:49 [hostname] account-server[182505]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdc/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0004 "-" 182505 -
Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdc (txn: tx9128250987cb4e07ad55d-00632c8d05)
Sep 22 12:27:49 [hostname] account-server[182515]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdj/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0004 "-" 182515 -
Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdj (txn: tx9128250987cb4e07ad55d-00632c8d05)
Sep 22 12:27:49 [hostname] account-server[182514]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdh/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182514 -
Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdh (txn: tx9128250987cb4e07ad55d-00632c8d05)
Sep 22 12:27:49 [hostname] account-server[182506]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdl/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182506 -
Sep 22 12:27:49 [hostname] account-server[182504]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdf/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0010 "-" 182504 -
Sep 22 12:27:49 [hostname] account-server[182504]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdi/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0002 "-" 182504 -
Sep 22 12:27:49 [hostname] proxy-server[182464]: Handoff requested (4) (txn: tx9128250987cb4e07ad55d-00632c8d05)
Sep 22 12:27:50 [hostname] account-server[182512]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sda/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182512 -
Sep 22 12:27:50 [hostname] proxy-server[182464]: Handoff requested (5) (txn: tx9128250987cb4e07ad55d-00632c8d05)
Sep 22 12:27:50 [hostname] account-server[182494]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sde/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0010 "-" 182494 -
Sep 22 12:27:50 [hostname] proxy-server[182464]: Handoff requested (6) (txn: tx9128250987cb4e07ad55d-00632c8d05)
Sep 22 12:27:50 [hostname] account-server[182506]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sdk/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0003 "-" 182506 -
Sep 22 12:27:50 [hostname] proxy-server[182464]: Account HEAD returning 503 for [507, 507, 507] (txn: tx9128250987cb4e07ad55d-00632c8d05)
Sep 22 12:27:50 [hostname] proxy-server[182464]: - - 22/Sep/2022/16/27/50 HEAD /v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8%3Fformat%3Djson HTTP/1.0 503 - Swift - - - - tx9128250987cb4e07ad55d-00632c8d05 - 0.0337 RL - 1663864069.976659298 1663864070.010398626 -
Sep 22 12:27:51 [hostname] proxy-server[182464]: [IP] 172.16.2.2 22/Sep/2022/16/27/51 HEAD /v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8%3Fformat%3Djson HTTP/1.0 403 - python-swiftclient-3.13.1 gAAAAABjLI0FH9MP... - - - tx9128250987cb4e07ad55d-00632c8d05 - 1.4298 - - 1663864069.970289707 1663864071.400121450 -
No data is being written to the disks by Swift:
[root at swift ~]# ls -lR /srv/node
/srv/node:
total 0
drwxr-xr-x. 2 swift swift 6 Sep 22 11:13 sda
drwxr-xr-x. 2 swift swift 6 Sep 22 11:07 sdb
drwxr-xr-x. 2 swift swift 6 Sep 22 11:07 sdc
...
/srv/node/sda:
total 0
/srv/node/sdb:
total 0
/srv/node/sdc:
total 0
...
What am I missing here?
More information about the openstack-discuss
mailing list