<div dir="ltr">Hi everyone,<div><br></div><div><div style="font-family:arial,sans-serif;font-size:13px">We're having a serious problem with our OpenStack installation. Most assets that we've been serving from <a href="http://assets00.grou.ps/" target="_blank">http://assets00.grou.ps</a> such as <a href="http://assets.grou.ps/0F2E3C/avatars/28346976/80.png" target="_blank">http://assets.grou.ps/0F2E3C/avatars/28346976/80.png</a> are broken, but some still remain intact: <a href="http://assets00.grou.ps/0F2E3C/userimages/gelistirme/20130830032959-ykqmvcnxaulxhcbwt-image50.png">http://assets00.grou.ps/0F2E3C/userimages/gelistirme/20130830032959-ykqmvcnxaulxhcbwt-image50.png</a></div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">Here's what happened:</div><div style="font-family:arial,sans-serif;font-size:13px"><br></div>
<div style="font-family:arial,sans-serif;font-size:13px">A few days ago, one of our servers crashed. The host (leaseweb) while rebooting the server for us (KVM access was broken as well) said that one of the disks was not being used and asked whether they should remove it. We said yes, do it. They did it and everything was fine until 12 hours ago.</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">I got a phone call saying all assets are broken. I checked the servers and saw that in 2 of our storage servers (we have 3 of them, each have 5 disks) -- the ones that they rebooted that day -- some of our disks were renamed, so old c0d5p1 and c0d6p1 were gone and now we had a c0d4p1 and c0d5p1. And in both servers c0d5p1 were full although they were supposed to be at 65%</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">My theory is, after they removed the disks, the disk names were reassigned and they changed fstab accordingly, but since the new naming didn't match our old configuration things went wrong. And perhaps most importantly, since c0d5p1 did exist before too, the system tried to recreate a partition so wrote things twice and that's why c0d5p1 was full but not others (c0d1p1, c0d2p1, c0d3p1 ..) </div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px">So I tried to fix this problem by unmounting c0d5p1 and c0d4p1 from these two servers and removing from the ring configuration. But things are still the same. Here's the message I receive when swift-ring-builder to remove and rebalance account/object/container.builder</div>
<div style="font-family:arial,sans-serif;font-size:13px"><br></div><div style="font-family:arial,sans-serif;font-size:13px"><div>root@proxy1:/etc/swift# swift-ring-builder account.builder rebalance</div><div>Reassigned 891289 (85.00%) partitions. Balance is now 33.33.</div>
<div>-------------------------------------------------------------------------------</div><div>NOTE: Balance of 33.33 indicates you should push this </div><div> ring, wait at least 1 hours, and rebalance/repush.</div>
<div>-------------------------------------------------------------------------------</div><div>root@proxy1:/etc/swift# swift-ring-builder account.builder </div><div>account.builder, build version 33</div><div>1048576 partitions, 3 replicas, 3 zones, 12 devices, 33.33 balance</div>
<div>The minimum number of hours before a partition can be reassigned is 1</div><div>Devices: id zone ip address port name weight partitions balance meta</div><div> 0 1 192.168.1.3 6002 c0d1p1 100.00 349525 33.33 </div>
<div> 1 1 192.168.1.3 6002 c0d2p1 100.00 349525 33.33 </div><div> 2 1 192.168.1.3 6002 c0d3p1 100.00 349526 33.33 </div><div> 3 2 192.168.1.4 6002 c0d1p1 100.00 262144 0.00 </div>
<div> 4 2 192.168.1.4 6002 c0d2p1 100.00 262144 0.00 </div><div> 5 2 192.168.1.4 6002 c0d3p1 100.00 262144 0.00 </div><div> 6 3 192.168.1.5 6002 c0d1p1 100.00 209715 -20.00 </div>
<div> 7 3 192.168.1.5 6002 c0d2p1 100.00 209716 -20.00 </div><div> 8 3 192.168.1.5 6002 c0d3p1 100.00 209715 -20.00 </div><div> 10 2 192.168.1.4 6002 c0d5p1 100.00 262144 0.00 </div>
<div> 11 3 192.168.1.5 6002 c0d5p1 100.00 209715 -20.00 </div><div> 14 3 192.168.1.5 6002 c0d6p1 100.00 209715 -20.00 </div><div><br></div><div>So I tried it once again 15 minutes ago and here's what I get:</div>
<div><br></div><div><div>root@proxy1:/etc/swift# swift-ring-builder account.builder rebalance</div><div>Cowardly refusing to save rebalance as it did not change at least 1%.</div><div>root@proxy1:/etc/swift# swift-ring-builder account.builder </div>
<div>account.builder, build version 33</div><div>1048576 partitions, 3 replicas, 3 zones, 12 devices, 33.33 balance</div><div>The minimum number of hours before a partition can be reassigned is 1</div><div>Devices: id zone ip address port name weight partitions balance meta</div>
<div> 0 1 192.168.1.3 6002 c0d1p1 100.00 349525 33.33 </div><div> 1 1 192.168.1.3 6002 c0d2p1 100.00 349525 33.33 </div><div> 2 1 192.168.1.3 6002 c0d3p1 100.00 349526 33.33 </div>
<div> 3 2 192.168.1.4 6002 c0d1p1 100.00 262144 0.00 </div><div> 4 2 192.168.1.4 6002 c0d2p1 100.00 262144 0.00 </div><div> 5 2 192.168.1.4 6002 c0d3p1 100.00 262144 0.00 </div>
<div> 6 3 192.168.1.5 6002 c0d1p1 100.00 209715 -20.00 </div><div> 7 3 192.168.1.5 6002 c0d2p1 100.00 209716 -20.00 </div><div> 8 3 192.168.1.5 6002 c0d3p1 100.00 209715 -20.00 </div>
<div> 10 2 192.168.1.4 6002 c0d5p1 100.00 262144 0.00 </div><div> 11 3 192.168.1.5 6002 c0d5p1 100.00 209715 -20.00 </div><div> 14 3 192.168.1.5 6002 c0d6p1 100.00 209715 -20.00 </div>
</div><div><br></div><div>Your help is highly appreciated. Thanks,</div><div><br></div><div>--</div><div>Emre</div><div><br></div></div></div><font face="yw-aa4fb83139ac691cb05d5cfd95da4950bbf70a7d-5e4a10713d6b10cd21c774197131e179--to" style></font><img src="https://app.yesware.com/t/aa4fb83139ac691cb05d5cfd95da4950bbf70a7d/5e4a10713d6b10cd21c774197131e179/spacer.gif" style="border: 0; width: 1px; height: 1px;"><img src="http://app.yesware.com/t/aa4fb83139ac691cb05d5cfd95da4950bbf70a7d/5e4a10713d6b10cd21c774197131e179/spacer.gif" style="border: 0; width: 1px; height: 1px;"></div>