Open Stack

Hello,

We started using EC policies in a new cluster a few months ago and
added more capacity. During the rebalance (started June 30), it seems
that all the data was copied to the new locations but it didn't clean
up the old locations. This was identified through our handoff
monitoring.

OS: Ubuntu 18.04
Swift: 2.17.1

Example:
List of devices for partition

~$ swift-get-nodes /etc/swift/object-4.ring.gz -p 14242
... removed ...
Server:Port Device      x.x.x.31:6031 d31
Server:Port Device      x.x.x.66:6030 d30
Server:Port Device      x.x.x.25:6029 d29
Server:Port Device      x.x.x.33:6027 d27
Server:Port Device      x.x.x.36:6020 d20
Server:Port Device      x.x.x.29:6018 d18
Server:Port Device      x.x.x.21:6033 d33
Server:Port Device      x.x.x.27:6025 d25
Server:Port Device      x.x.x.35:6022 d22
Server:Port Device      x.x.x.39:6031 d31
Server:Port Device      x.x.x.28:6032 d32
Server:Port Device      x.x.x.23:6021 d21
Server:Port Device      x.x.x.26:6022 d22
Server:Port Device      x.x.x.34:6023 d23
Server:Port Device      x.x.x.37:6019 d19
Server:Port Device      x.x.x.30:6017 d17
Server:Port Device      x.x.x.22:6027 d27
Server:Port Device      x.x.x.24:6031 d31
Server:Port Device      x.x.x.32:6032 d32

Partitions look to have the correct data on them:

~$ ssh root at x.x.x.31 "ls -lah ${DEVICE:-/srv/node*}/d31/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.66 "ls -lah ${DEVICE:-/srv/node*}/d30/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.25 "ls -lah ${DEVICE:-/srv/node*}/d29/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.33 "ls -lah ${DEVICE:-/srv/node*}/d27/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.36 "ls -lah ${DEVICE:-/srv/node*}/d20/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.29 "ls -lah ${DEVICE:-/srv/node*}/d18/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.21 "ls -lah ${DEVICE:-/srv/node*}/d33/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.27 "ls -lah ${DEVICE:-/srv/node*}/d25/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.35 "ls -lah ${DEVICE:-/srv/node*}/d22/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.39 "ls -lah ${DEVICE:-/srv/node*}/d31/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.28 "ls -lah ${DEVICE:-/srv/node*}/d32/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.23 "ls -lah ${DEVICE:-/srv/node*}/d21/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.26 "ls -lah ${DEVICE:-/srv/node*}/d22/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.34 "ls -lah ${DEVICE:-/srv/node*}/d23/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.37 "ls -lah ${DEVICE:-/srv/node*}/d19/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.30 "ls -lah ${DEVICE:-/srv/node*}/d17/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.22 "ls -lah ${DEVICE:-/srv/node*}/d27/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.24 "ls -lah ${DEVICE:-/srv/node*}/d31/objects-4/14242 | wc -l"
664
~$ ssh root at x.x.x.32 "ls -lah ${DEVICE:-/srv/node*}/d32/objects-4/14242 | wc -l"
664

>From one of the nodes that does not belong to the list above. This
partition should not exist on this node after the rebalance.

x.x.x.20:~# ls /srv/node/d28/objects-4/14242 | wc -l
627

The reconstructor is throwing a lot of these unexpected response
errors in the logs. Manually running it from the node that should not
have the partition, I can reproduce the error. x.x.y.0/24 is the
replication network.

x.x.x.20:~# swift-object-reconstructor /etc/swift/object-server.conf
-d d28 -p 14242 -o -v
object-reconstructor: x.x.y.42:6200/d30/14242 Unexpected response:
":ERROR: 500 'ERROR: With :UPDATES: 36 failures to 0 successes'"

There looked to have been some partition locations cleaned up around
July 11. Our expectation is that the old partition locations should be
cleaned gradually since June 30 but we're not seeing that.
I Was hoping for some ideas on what may be the problem (if any) and
how we can make sure that the old partitions are cleaned up.

Thanks,
Reid