Open Stack

On 10/12/2017 05:52 PM, Ian Wienand wrote:
> I tried this in order, firstly recreating references.db (didn't help)
> and so I have started the checksums.db recreation.  This is now
> running; I just moved the old one out of the way

Well, that didn't go so well.  The output flooded stuff and then it
died.

---
...
Within references.db subtable references at get: No such file or directory
BDB0134 read: 0x11989b0, 4096: No such file or directory
Internal error of the underlying BerkeleyDB database:
Within references.db subtable references at get: No such file or directory
BDB0134 read: 0x11989b0, 4096: No such file or directory
Internal error of the underlying BerkeleyDB database:
Within references.db subtable references at get: No such file or directory
37 files were added but not used.
The next deleteunreferenced call will delete them.
BDB0151 fsync: Connection timed out
BDB0164 close: Connection timed out
./db/checksums.db: Connection timed out
BDB3028 ./db/checksums.db: unable to flush: Connection timed out
db_close(checksums.db, pool): Connection timed out
Error creating './db/version.new': Connection timed out(errno is 110)
Error 110 deleting lock file './db/lockfile': Connection timed out!
There have been errors!
---

Presumably this matches up with the AFS errors logged

---
[Thu Oct 12 09:19:59 2017] afs: Lost contact with file server 104.130.138.161 in cell openstack.org (code -512) (all multi-homed ip addresses down for the server)
[Thu Oct 12 09:19:59 2017] afs: Lost contact with file server 104.130.138.161 in cell openstack.org (code -512) (all multi-homed ip addresses down for the server)
[Thu Oct 12 09:19:59 2017] afs: failed to store file (110)
[Thu Oct 12 09:20:02 2017] afs: failed to store file (110)
[Thu Oct 12 09:20:10 2017] afs: file server 104.130.138.161 in cell openstack.org is back up (code 0) (multi-homed address; other same-host interfaces may still be down)
[Thu Oct 12 09:20:10 2017] afs: file server 104.130.138.161 in cell openstack.org is back up (code 0) (multi-homed address; other same-host interfaces may still be down)
---

I restarted for good luck, but if this is transient network issues, I
guess it will just happen again.  ping shows no packet loss, but very
occasional latency spikes, fwiw.

We restarted mirror-update; maybe it's worth restarting the AFS
servers too?

-i