Swift not recognizing mounted drives
Hi all. I have a new Openstack Swift install, currently a single node with 12 disks running CentOS 8 Stream and Openstack Yoga / RDO release with Keystone Auth. Disks are formatted with xfs and mounted at /srv/node/sd[a-l] as described in the Install docs, but no Swift services reccognize it. SELinux is set to permissive mode. Log sample: Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sda is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdb is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdc is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdd is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sde is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdf is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdg is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdh is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdi is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdj is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdk is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Skipping: /srv/node/sdl is not mounted Sep 22 11:29:01 [hostname] account-replicator[182475]: Beginning replication run Sep 22 11:29:01 [hostname] account-replicator[182475]: Replication run OVER Sep 22 11:29:01 [hostname] account-replicator[182475]: Attempted to replicate 0 dbs in 0.00344 seconds (0.00000/s) Sep 22 11:29:01 [hostname] account-replicator[182475]: Removed 0 dbs Sep 22 11:29:01 [hostname] account-replicator[182475]: 0 successes, 144 failures Sep 22 11:29:01 [hostname] account-replicator[182475]: diff:0 diff_capped:0 empty:0 hashmatch:0 no_change:0 remote_merge:0 rsync:0 ts_repl:0 Container and Object processes show the same issue. And yet...devices are mounted: [root@swift]# mount | grep srv /dev/sdb on /srv/node/sdb type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdc on /srv/node/sdc type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdd on /srv/node/sdd type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sde on /srv/node/sde type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdf on /srv/node/sdf type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdg on /srv/node/sdg type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdh on /srv/node/sdh type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdi on /srv/node/sdi type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdj on /srv/node/sdj type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdk on /srv/node/sdk type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sdl on /srv/node/sdl type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) /dev/sda on /srv/node/sda type xfs (rw,noatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota) And the rings are built: [root@swift]# swift-ring-builder /etc/swift/account.builder /etc/swift/account.builder, build version 13, id 415987b4753847e9924147119872d547 2048 partitions, 3.000000 replicas, 1 regions, 1 zones, 12 devices, 0.00 balance, 0.00 dispersion The minimum number of hours before a partition can be reassigned is 1 (0:16:41 remaining) The overload factor is 0.00% (0.000000) Ring file /etc/swift/account.ring.gz is up-to-date Devices: id region zone ip address:port replication ip:port name weight partitions balance flags meta 0 1 1 172.16.2.2:6202 172.16.2.2:6202 sda 100.00 512 0.00 1 1 1 172.16.2.2:6202 172.16.2.2:6202 sdb 100.00 512 0.00 2 1 1 172.16.2.2:6202 172.16.2.2:6202 sdc 100.00 512 0.00 3 1 1 172.16.2.2:6202 172.16.2.2:6202 sdd 100.00 512 0.00 4 1 1 172.16.2.2:6202 172.16.2.2:6202 sde 100.00 512 0.00 5 1 1 172.16.2.2:6202 172.16.2.2:6202 sdf 100.00 512 0.00 6 1 1 172.16.2.2:6202 172.16.2.2:6202 sdg 100.00 512 0.00 7 1 1 172.16.2.2:6202 172.16.2.2:6202 sdh 100.00 512 0.00 8 1 1 172.16.2.2:6202 172.16.2.2:6202 sdi 100.00 512 0.00 9 1 1 172.16.2.2:6202 172.16.2.2:6202 sdj 100.00 512 0.00 10 1 1 172.16.2.2:6202 172.16.2.2:6202 sdk 100.00 512 0.00 11 1 1 172.16.2.2:6202 172.16.2.2:6202 sdl 100.00 512 0.00 Bash certainly thinks they are actually mounted and writable: [root@swift ~]# ls -l /srv/node/sda total 0 [root@swift ~]# touch /srv/node/sda/file [root@swift ~]# ls /srv/node/sda file [root@swift ~]# umount /dev/sda [root@swift ~]# ls /srv/node/sda [root@swift ~]# mount /srv/node/sda/ [root@swift ~]# ls /srv/node/sda file [root@swift ~]# rm /srv/node/sda/file rm: remove regular empty file '/srv/node/sda/file'? y And Swift is configured to look at /srv/node for the devices: [root@swift ~]# grep devices /etc/swift/*.conf /etc/swift/account-server.conf:devices = /srv/node /etc/swift/container-server.conf:devices = /srv/node /etc/swift/object-server.conf:devices = /srv/node Running "swift info" shows the expected configuration output, but "swift stat" throws this error: [root@controller ~]# swift stat Account HEAD failed: https:/[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8 403 Forbidden Failed Transaction ID: tx9128250987cb4e07ad55d-00632c8d05 and the logs show account-server throwing "507 ERROR Insufficient Storage" before 503, and then returning a 403. [root@swift ~]# journalctl -b | grep tx9128250987cb4e07ad55d-00632c8d05 Sep 22 12:27:49 [hostname] account-server[182505]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdc/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0004 "-" 182505 - Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdc (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:49 [hostname] account-server[182515]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdj/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0004 "-" 182515 - Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdj (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:49 [hostname] account-server[182514]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdh/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182514 - Sep 22 12:27:49 [hostname] proxy-server[182464]: ERROR Insufficient Storage 172.16.2.2:6202/sdh (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:49 [hostname] account-server[182506]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdl/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182506 - Sep 22 12:27:49 [hostname] account-server[182504]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdf/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0010 "-" 182504 - Sep 22 12:27:49 [hostname] account-server[182504]: 172.16.2.2 - - [22/Sep/2022:16:27:49 +0000] "HEAD /sdi/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0002 "-" 182504 - Sep 22 12:27:49 [hostname] proxy-server[182464]: Handoff requested (4) (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] account-server[182512]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sda/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0011 "-" 182512 - Sep 22 12:27:50 [hostname] proxy-server[182464]: Handoff requested (5) (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] account-server[182494]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sde/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0010 "-" 182494 - Sep 22 12:27:50 [hostname] proxy-server[182464]: Handoff requested (6) (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] account-server[182506]: 172.16.2.2 - - [22/Sep/2022:16:27:50 +0000] "HEAD /sdk/456/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8" 507 - "HEAD http://[api endpoint]/v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8?format=json" "tx9128250987cb4e07ad55d-00632c8d05" "proxy-server 182464" 0.0003 "-" 182506 - Sep 22 12:27:50 [hostname] proxy-server[182464]: Account HEAD returning 503 for [507, 507, 507] (txn: tx9128250987cb4e07ad55d-00632c8d05) Sep 22 12:27:50 [hostname] proxy-server[182464]: - - 22/Sep/2022/16/27/50 HEAD /v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8%3Fformat%3Djson HTTP/1.0 503 - Swift - - - - tx9128250987cb4e07ad55d-00632c8d05 - 0.0337 RL - 1663864069.976659298 1663864070.010398626 - Sep 22 12:27:51 [hostname] proxy-server[182464]: [IP] 172.16.2.2 22/Sep/2022/16/27/51 HEAD /v1/AUTH_5198a36c0f454ae49e9e8a5d6e154ee8%3Fformat%3Djson HTTP/1.0 403 - python-swiftclient-3.13.1 gAAAAABjLI0FH9MP... - - - tx9128250987cb4e07ad55d-00632c8d05 - 1.4298 - - 1663864069.970289707 1663864071.400121450 - No data is being written to the disks by Swift: [root@swift ~]# ls -lR /srv/node /srv/node: total 0 drwxr-xr-x. 2 swift swift 6 Sep 22 11:13 sda drwxr-xr-x. 2 swift swift 6 Sep 22 11:07 sdb drwxr-xr-x. 2 swift swift 6 Sep 22 11:07 sdc ... /srv/node/sda: total 0 /srv/node/sdb: total 0 /srv/node/sdc: total 0 ... What am I missing here?
On Thu, Sep 22, 2022 at 12:02 PM Andrew Boring <andrew@andrewboring.com> wrote:
What am I missing here?
Wow I wish I knew! It looks like you tried everything. ??? I have a troubleshooting strategy I call "start with something that works" - the 507 in the logs makes it look like the device mount check is failing for some reason - but you can disable that: https://opendev.org/openstack/swift/src/branch/master/etc/account-server.con... Does it "work" if we turn off the mount check entirely? Have you been able to configure swift successfully in the past on *different* hardware or software configuration(s)? -- Clay Gerrard
Hi Clay. This is my first Swift install from Openstack packages. My prior expeirence with Swift has been through third-party systems that abstract a lot of the day-to-day disk mgmt activities, so this my first run-through manually. My personal installs have always been on Supermicro hardware, so I don't think that would be an issue (standard SATA controller, not RAID, etc), though I'm happy to check anything further. But if the kernel and shell are seeing the mounted disks without issue, I have to think it's something at the Swift process level. I did try disabling mount_check, which gives me this instead: Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sda is not a directory Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sdb is not a directory Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sdc is not a directory Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sdd is not a directory Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sde is not a directory Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sdf is not a directory Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sdg is not a directory Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sdh is not a directory Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sdi is not a directory Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sdj is not a directory Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sdk is not a directory Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sdl is not a directory Sep 22 13:12:46 [hostname] account-replicator[183238]: Beginning replication run Sep 22 13:12:46 [hostname] account-replicator[183238]: Replication run OVER Sep 22 13:12:46 [hostname] account-replicator[183238]: Attempted to replicate 0 dbs in 0.00372 seconds (0.00000/s) Sep 22 13:12:46 [hostname] account-replicator[183238]: Removed 0 dbs Sep 22 13:12:46 [hostname] account-replicator[183238]: 0 successes, 144 failures Sep 22 13:12:46 [hostname] account-replicator[183238]: diff:0 diff_capped:0 empty:0 hashmatch:0 no_change:0 remote_merge:0 rsync:0 ts_repl:0 Sep 22 13:12:47 [hostname] object-replicator[183230]: Starting object replication pass. And yet: [root@swift]# file /srv/node/sda /srv/node/sda: directory [root@swift]# file /srv/node/sdb /srv/node/sdb: directory ...and so on. Running "swift stat" with mount_check = false gives the same 507 Insufficient Storage as before. -Andrew
On Sep 22, 2022, at 1:08 PM, Clay Gerrard <clay.gerrard@gmail.com> wrote:
On Thu, Sep 22, 2022 at 12:02 PM Andrew Boring <andrew@andrewboring.com> wrote:
What am I missing here?
Wow I wish I knew! It looks like you tried everything. ???
I have a troubleshooting strategy I call "start with something that works" - the 507 in the logs makes it look like the device mount check is failing for some reason - but you can disable that:
https://opendev.org/openstack/swift/src/branch/master/etc/account-server.con...
Does it "work" if we turn off the mount check entirely? Have you been able to configure swift successfully in the past on *different* hardware or software configuration(s)?
-- Clay Gerrard
On Thu, Sep 22, 2022 at 12:36 PM Andrew Boring <andrew@andrewboring.com> wrote:
Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sda is not a directory
Ok well that's right here: https://opendev.org/openstack/swift/src/branch/master/swift/common/db_replic... Which you can look at in constraints is pretty straightforward: https://opendev.org/openstack/swift/src/branch/master/swift/common/constrain...
[root@swift]# file /srv/node/sda /srv/node/sda: directory
ok, that seems like something that works - let's see if we can get closer to the problem? What if we use python (as root): sudo python -c "import os.path; print(os.path.isdir('/srv/node/sda'))" ... or again with the "swift" user sudo -u swift python -c "import os.path; print(os.path.isdir('/srv/node/sda'))" N.B. the user swift is just the default - use whatever you have set as "user =" in your config Also check syslog/journald and security or whatever for any warning logs about permissions or SELinux -- Clay Gerrard
Clay, Well, I'm an idiot. I absolutely remembered to cchange the SELinux config file to permissive, and absolutely forgot to 'setenforce 0' immediately afteward. I even watched the audit log, thinking it was just logging the actions that will be blocked when I set it back to enforcing. [root@swift ~]# grep SELINUX= /etc/selinux/config # SELINUX= can take one of these three values: SELINUX=permissive And yet: [root@swift ~]# getenforce Enforcing A quick "setenforce 0" made all the nasty errors go away. Thanks for your patience.
On Sep 22, 2022, at 3:06 PM, Clay Gerrard <clay.gerrard@gmail.com> wrote:
On Thu, Sep 22, 2022 at 12:36 PM Andrew Boring <andrew@andrewboring.com> wrote:
Sep 22 13:12:46 [hostname] account-replicator[183238]: Skipping: /srv/node/sda is not a directory
Ok well that's right here:
https://opendev.org/openstack/swift/src/branch/master/swift/common/db_replic...
Which you can look at in constraints is pretty straightforward:
https://opendev.org/openstack/swift/src/branch/master/swift/common/constrain...
[root@swift]# file /srv/node/sda /srv/node/sda: directory
ok, that seems like something that works - let's see if we can get closer to the problem? What if we use python (as root):
sudo python -c "import os.path; print(os.path.isdir('/srv/node/sda'))"
... or again with the "swift" user
sudo -u swift python -c "import os.path; print(os.path.isdir('/srv/node/sda'))"
N.B. the user swift is just the default - use whatever you have set as "user =" in your config
Also check syslog/journald and security or whatever for any warning logs about permissions or SELinux
-- Clay Gerrard
participants (2)
-
Andrew Boring
-
Clay Gerrard