Ok, if I was giving out t-shirts for finding this issue then the prize would go to Pete. Thank you!!!!<div><br></div><div>Disabling fallocate did the trick. I was slowly working my way through all the object-server config options and hadn't gotten to that one yet. Turning features on and off by brute force is admittedly lame, but sometimes that's all you have.</div>
<div><br></div><div>I also turned off all the other things I was doing to try to slow down the mini-servers, but disabling fallocate was all that was necessary. Here is my config:</div><div><br></div><div><div>[DEFAULT]</div>
<div>bind_ip = 192.168.1.202</div><div>workers = 1</div><div>disable_fallocate = true</div><div><br></div><div>[pipeline:main]</div><div>pipeline = object-server</div><div><br></div><div>[app:object-server]</div><div>use = egg:swift#object</div>
<div><br></div><div>[object-replicator]</div><div><br></div><div>[object-updater]</div><div><br></div><div>[object-auditor]</div><div><br></div><div>A few more details... </div><div><br></div><div>My servers are running Ubuntu 12.04 LTS. A straight-up apt-get of all the pre-requisites did NOT produce a working Swift deployment on Arm. Although all the dependencies would deploy fine and the Swift services would start up, the proxy-server could not communicate with the storage nodes.</div>
<div><br></div><div>So I also had to get older, Armel versions of the python-greenlet and python-eventlet. </div><div><br></div><div>
<a href="https://launchpad.net/ubuntu/precise/armel/python-greenlet/0.3.1-1ubuntu5.1" style="font-family:Arial">https://launchpad.net/ubuntu/precise/armel/python-greenlet/0.3.1-1ubuntu5.1</a><div style="font-family:Arial">
<a href="https://launchpad.net/ubuntu/precise/armel/python-eventlet/0.9.16-1ubuntu4.1">https://launchpad.net/ubuntu/precise/armel/python-eventlet/0.9.16-1ubuntu4.1</a></div></div><div><br></div><div>Once I deployed those older libraries for Armel, then my Swift cluster worked (except for the fallocate issue).</div>
<div><br></div><div>Thanks for everyone's help.</div><div><br></div><div>-N</div><div><br></div><div class="gmail_quote">On Tue, Oct 30, 2012 at 11:07 AM, Nathan Trueblood <span dir="ltr"><<a href="mailto:nathan@truebloodllc.com" target="_blank">nathan@truebloodllc.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">The filesystem is XFS, and I used the recommended mkfs and mount options for Swift.<div><br></div><div>The file size seems to have no bearing on the issue, although I haven't tried really tiny files. Bigfile3 is only 200K.</div>
<div><br></div><div>I'll try disabling fallocate...<div><div class="h5"><br><br><div class="gmail_quote">On Mon, Oct 29, 2012 at 7:37 PM, Pete Zaitcev <span dir="ltr"><<a href="mailto:zaitcev@redhat.com" target="_blank">zaitcev@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>On Mon, 29 Oct 2012 18:16:52 -0700<br>
Nathan Trueblood <<a href="mailto:nathan@truebloodllc.com" target="_blank">nathan@truebloodllc.com</a>> wrote:<br>
<br>
> Definitely NOT a problem with the filesystem, but something is causing the<br>
> object-server to think there is a problem with the filesystem.<br>
<br>
</div>If you are willing to go all-out, you can probably catch the<br>
error with strace, if it works on ARM. Failing that, find all places<br>
where 507 is generated and see if any exceptions are caught, by<br>
modifying the source, I'm afraid to say.<br>
<div><br>
> I suspect a bug in one of the underlying libraries.<br>
<br>
</div>That's a possibility. Or, it could be a kernel bug. You are using XFS,<br>
right? If it were something other than XFS or ext4, I would suspect<br>
ARM blowing over the 2GB barrier somewhere, since your object is<br>
called "bigfile3". As it is, you have little option than to divide<br>
the layers until you identify the one that's broken.<br>
<br>
BTW, make sure to disable the fallocate, since we're at it.<br>
<span><font color="#888888"><br>
-- Pete<br>
</font></span></blockquote></div><br></div></div></div>
</blockquote></div><br></div>