<div dir="ltr">Run `swift-ring-builder /etc/swift/object.builder validate` - it should have no errors and exit 0. Can you provide a paste of the output from `swift-ring-builder /etc/swift/object.builder` as well - it should list some general info about the ring (number of replicas, and list of devices). Rebalance the ring and make sure it's been distributed to all nodes.<div>
<br></div><div>The particular line you're seeing pop up in the traceback seems to be looking for all of the nodes for a particular partition it found in the objects' dir. I'm not seeing any local sanitization [1] around those top level directory names, so maybe it's just some garbage that created there outside of swift, or some file system corruption?</div>
<div><br></div><div>Can you provide the output from `ls /srv/node/objects` (or wherever you have devices configured)</div><div><br></div><div>-Clay</div><div><br></div><div>1. <a href="https://bugs.launchpad.net/swift/+bug/1229372">https://bugs.launchpad.net/swift/+bug/1229372</a></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Sep 23, 2013 at 2:34 AM, Mike Preston <span dir="ltr"><<a href="mailto:mike.preston@synety.com" target="_blank">mike.preston@synety.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-GB" link="blue" vlink="purple">
<div>
<p class="MsoNormal">Hi, <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">We are seeing a replication error on swift. The error only is seen on a single node, the other nodes appear to be working fine.<u></u><u></u></p>
<p class="MsoNormal">Installed version is debian wheezy with swift 1.4.8-2+deb7u1
<br>
<br>
<u></u><u></u></p>
<p class="MsoNormal">Sep 23 10:33:03 storage-node-01 object-replicator Starting object replication pass.<u></u><u></u></p>
<p class="MsoNormal">Sep 23 10:33:03 storage-node-01 object-replicator Exception in top-level replication loop: #012Traceback (most recent call last):#012 File "/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line 564, in replicate#012 jobs =
self.collect_jobs()#012 File "/usr/lib/python2.7/dist-packages/swift/obj/replicator.py", line 536, in collect_jobs#012 self.object_ring.get_part_nodes(int(partition))#012 File "/usr/lib/python2.7/dist-packages/swift/common/ring/ring.py", line 103, in
get_part_nodes#012 return [self.devs[r[part]] for r in self._replica2part2dev_id]#012IndexError: array index out of range<u></u><u></u></p>
<p class="MsoNormal">Sep 23 10:33:03 storage-node-01 object-replicator Nothing replicated for 0.728466033936 seconds.<u></u><u></u></p>
<p class="MsoNormal">Sep 23 10:33:03 storage-node-01 object-replicator Object replication complete. (0.01 minutes)<br>
<br>
<u></u><u></u></p>
<p class="MsoNormal">Can anyone shed any light on this or next steps in debugging it or fixing it?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><b>Mike Preston<u></u><u></u></b></p>
<p class="MsoNormal">Infrastructure Team | SYNETY<u></u><u></u></p>
<p class="MsoNormal"><a href="http://www.synety.com" target="_blank"><span style="color:blue">www.synety.com</span></a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">direct: 0116 424 4016<u></u><u></u></p>
<p class="MsoNormal">mobile: 07950 892038<u></u><u></u></p>
<p class="MsoNormal">main: 0116 424 4000<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<br>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></blockquote></div><br></div>