<div dir="ltr">Further to above details <div>please find related logs ... </div><div><br></div><div><b><u>object server</u></b></div><div><br></div><div><div>4:07:21:20 +0000] "DELETE /vdb1/844/AUTH_215c458021024a8c87471157a7040644/suntmp7/objkeye385cf9c-ad96-423f-8c8d-94d001631f8b" 409 - "DELETE <a href="http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/suntmp7/objkeye385cf9c-ad96-423f-8c8d-94d001631f8b" target="_blank">http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/suntmp7/objkeye385cf9c-ad96-423f-8c8d-94d001631f8b</a>" "txcb3b4b47278e483ca560e-005350d271" "proxy-server 9619" 0.0004</div>
<div><br></div><div>or</div><div><br></div><div>[22/Apr/2014:08:30:07 +0000] "DELETE /vdb1/238/AUTH_215c458021024a8c87471157a7040644/zoom43/objkeyf03550a3-2be9-4b59-b12b-df75378aea14" 409 - "DELETE <a href="http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom43/objkeyf03550a3-2be9-4b59-b12b-df75378aea14">http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom43/objkeyf03550a3-2be9-4b59-b12b-df75378aea14</a>" "tx0bc79d62fec445d28dfab-005356288f" "proxy-server 2142" 0.0005<br>
</div>
</div><div><br></div><div><br></div><div>
</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Apr 23, 2014 at 8:20 AM, Sumit Gaur <span dir="ltr"><<a href="mailto:sumitkgaur@gmail.com" target="_blank">sumitkgaur@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Hi clay,<br>
Thanks for responding , first about setup , it is multi node env, and problem is with delete object .</p>
<p dir="ltr"> I checked proxy and storage nodes clock, they are in sync. Also storage node run only main daemons I.e. no auditor, and others. </p>
<p dir="ltr">I traced the 409 call and it is coming from object server.<br>
I am generating random id for key so no reuse of same name. But after one put(t1) next is del(t1).....is this could be a problem...I think they are sync calls.</p>
<p dir="ltr">Regards<span class="HOEnZb"><font color="#888888"><br>
Sumit<br></font></span></p>
<p dir="ltr"></p><div class="">On Apr 23, 2014 5:14 AM, "Clay Gerrard" <<a href="mailto:clay.gerrard@gmail.com" target="_blank">clay.gerrard@gmail.com</a>> wrote:<br>
><br>
> 409 on DELETE (object?) is a pretty specific error. That should mean that the timestamp assigned to the delete is earlier than the timestamp of the data file.<br>
><br>
> Most likely mean that you're getting some time-drift on your proxies (but that assumes multi-node) or maybe that you're reusing names between threads and your object server's see PUT(ts1) PUT(ts3) DELETE(ts2) - but that'd be a pretty tight race...<br>
><br>
> Should all be logged - try and find a DELETE that went 409 and trace the transaction id.<br>
><br>
><br>
> On Mon, Apr 21, 2014 at 5:54 AM, taurus huang <<a href="mailto:huanggeng.8552@gmail.com" target="_blank">huanggeng.8552@gmail.com</a>> wrote:<br>
>><br>
>> Please provide the log file： /var/log/swift/swift.log AND /var/log/keystone/keystone.log<br>
>><br>
>><br>
>> On Mon, Apr 21, 2014 at 11:55 AM, Sumit Gaur <<a href="mailto:sumitkgaur@gmail.com" target="_blank">sumitkgaur@gmail.com</a>> wrote:<br>
>>><br>
>>> Hi <br>
>>> I using jclouds lib integrated with Openstack Swift+ keystone combination. Things are working fine except stability test. After 20-30 hours of test jclouds/SWIFT start degrading in TPS and keep going down over the time. <br>
>>><br>
>>> 1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads.<br>
>>> 2) I am getting a lot of 409 and DEL failure for the response too from SWIFT.<br>
>>><br>
>>><br>
>>> Can sombody help me what is going wrong here ?<br>
>>><br>
>>> Thanks<br></div><div class="">
>>> sumit<br>
>>><br>
>>> _______________________________________________<br>
>>> OpenStack-dev mailing list<br>
>>> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
</div><p></p>
</blockquote></div><br></div>