<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<font size="-1"><font face="Helvetica, Arial, sans-serif">Hi All,<br>
<br>
After few more days of checks I couldn't resolve it. I think I
tried pretty much everything.<br>
<br>
I tend to think that the problem is CentOS/RHEL 5.x and the
following error:<br>
<br>
Dec 22 05:03:24 ec01 object-server STDOUT:
WARNING:root:Unable to locate fallocate in libc. Leaving as a
no-op.<br>
<br>
I think the lack of fallocate() in EL5 kernel hinders sequential
</font></font><font size="-1"><font face="Helvetica, Arial,
sans-serif">read/</font></font><font size="-1"><font
face="Helvetica, Arial, sans-serif">write operation. When
multiple streams are writing data (3x object data + local and
remote container updates), disk i/o degrades significantly. I
tried to fix it by replacing fallocate() with posix_fallocate()
( <a class="moz-txt-link-freetext" href="http://paste.openstack.org/show/3932/">http://paste.openstack.org/show/3932/</a> ), but things gone even
worse. This is my theory and unfortunately I don't have a way to
prove it.<br>
<br>
Based on that, I think authors need to make clear that Swift is
not supported on EL5 and minimum version should be EL6 (where
fallocate() exists) or fix it in swift.<br>
<br>
<br>
Regards,<br>
Rustam.<br>
<br>
</font></font><br>
On 20/12/2011 13:59, Ywang225 wrote:
<blockquote cite="mid:34D14DA6-9125-4AB5-9CB1-C18C7C2D76A9@126.com"
type="cite">
<div>Both Metadata.update() and container_update() should apply on
all replicas, and metadata.update() will write xattrs, and
container_update() will write to sqlite db files. Surprisingly
to see so long time to update both.</div>
<div><br>
</div>
<div>--ywang</div>
<div><br>
发自我的 iPhone</div>
<div><br>
在 2011-12-20，19:13，Rustam Aliyev <<a moz-do-not-send="true"
href="mailto:rustam@code.az">rustam@code.az</a>> 写道：<br>
<br>
</div>
<blockquote type="cite">
<div> <font size="-1"><font face="Helvetica, Arial, sans-serif">Hi
Mike,<br>
<br>
Thanks, I didn't know that PUT operation also includes
updating replica containers. That makes sense. I will
check that.<br>
<br>
In the mean time I've added debug checkpoints into PUT
operation to measure different steps. Modified code is
here: <a moz-do-not-send="true"
href="http://paste.openstack.org/show/3899/">http://paste.openstack.org/show/3899/</a>
(original: <a moz-do-not-send="true"
href="https://github.com/openstack/swift/blob/master/swift/obj/server.py#L530">https://github.com/openstack/swift/blob/master/swift/obj/server.py#L530</a>).
Basically, I added some self.logger.debug() with timestamp
in few places.<br>
<br>
I'm not python dev and don't know swift internals, so it's
quite possible that I've got something wrong. In any case
here are few sample results: <a moz-do-not-send="true"
href="http://paste.openstack.org/show/3900/">http://paste.openstack.org/show/3900/</a><br>
<br>
Basically, there are 2 steps which take too long:<br>
</font></font>
<ol>
<li>write metadata (line #63 of the first paste,
metadata.update()), this takes 0.5-1.0 sec!</li>
<li>update container (line #85, self.container_update()),
this is second slowest, also in the range of ~0.5-1.0 sec.</li>
</ol>
I assume that self.container_update() includes replica
updates. But why does metadata.update() takes so long? Does it
also imply replica updates?<br>
<br>
Overall, I have to say that troubleshooting of swift is
impossible. There's almost no difference between log levels
INFO and DEBUG. Would be nice to have some more info in DEBUG
and even TRACE level for this kinda problems.<br>
<br>
--<br>
Rustam.<br>
<br>
On 20/12/2011 04:41, Michael Barton wrote:
<blockquote
cite="mid:CADu4cP2zkz_Sq7LqGrd2E3cu7B5T=doyMdB7yOreZmQ0O0nx6g@mail.gmail.com"
type="cite">
<pre wrap="">On Mon, Dec 19, 2011 at 6:21 AM, Rustam Aliyev <a moz-do-not-send="true" class="moz-txt-link-rfc2396E" href="mailto:rustam@code.az"><</a><a moz-do-not-send="true" href="mailto:rustam@code.az">rustam@code.az</a>> wrote:
</pre>
<blockquote type="cite">
<pre wrap="">The only thing which looks suspicious to me are these errors:
Dec 18 04:01:28 ec01 object-server ERROR container update failed with
10.0.1.3:6001/d01 (saving for async update later): Timeout (3s) (txn:
txdf95ad5a10844ee0b74d70d8a7638082)
Dec 18 04:01:28 ec01 object-server ERROR container update failed with
10.0.1.2:6001/d01 (saving for async update later): Timeout (3s) (txn:
txee2545ba4610430fa3a6a166ca50c574)
Dec 18 04:01:28 ec01 object-server ERROR container update failed with
10.0.1.8:6001/d01 (saving for async update later): Timeout (3s) (txn:
tx2546b29b15c643ec90a122a753dfddd3)
</pre>
</blockquote>
<pre wrap="">Yeah, that is likely to be the culprit. Each write is taking at least
3 seconds because it's timing out trying to update the container
servers.
So you need to debug connectivity from this object server to those IP
addresses on port 6001 -- that the IP addresses and port are correct,
everything's on the same network, there aren't any firewall rules
blocking those connections, that the container servers are running and
accepting connections, etc. I'll read through your paste in a bit and
see if I notice anything.
-- Mike
</pre>
</blockquote>
</div>
</blockquote>
<blockquote type="cite">
<div><span>_______________________________________________</span><br>
<span>Mailing list: <a moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack">https://launchpad.net/~openstack</a></span><br>
<span>Post to : <a moz-do-not-send="true"
href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a></span><br>
<span>Unsubscribe : <a moz-do-not-send="true"
href="https://launchpad.net/%7Eopenstack">https://launchpad.net/~openstack</a></span><br>
<span>More help : <a moz-do-not-send="true"
href="https://help.launchpad.net/ListHelp">https://help.launchpad.net/ListHelp</a></span><br>
</div>
</blockquote>
</blockquote>
</body>
</html>