<div dir="ltr"><div><b>FYI more concrete logs </b><br></div><div><b><br></b></div><div><b>Client Node</b></div><div>Request failed for objkey objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615 org.jclouds.http.HttpResponseException: command: DELETE <a href="http://www.example.com:80/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615">http://www.example.com:80/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615</a> HTTP/1.1 failed with response: HTTP/1.1 409 Conflict; content: [<html><h1>Conflict</h1><p>There was a conflict when trying to complete your request.</p></html>]</div>
<div><br></div><div><b>Proxy Node </b></div><div>Apr 23 12:00:27 proxy1 swift-proxy allow user with role admin as account admin (txn: txe45b336b0ab44aee8a698-0053572ccb) (client_ip: 10.41.54.27)</div><div><br></div><div>
<br>
</div><div><b>Storage Node</b></div><div>Apr 23 12:00:26 object1<b> container-server</b> 10.41.54.9 - - [23/Apr/2014:03:00:26 +0000] "PUT /vdb1/955/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615" 201 - "tx3ba78bc8924349d18f7af-0053572ccc" "PUT <a href="http://www.example.com/vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615">http://www.example.com/vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615</a>" "obj-server 17908" 0.0006</div>
<div><br></div><div>Apr 23 12:00:26 object1 <b>object-server</b> 10.41.54.21 - - [23/Apr/2014:03:00:26 +0000] "PUT /vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615" 201 - "<b>PUT </b><a href="http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615">http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615</a>" "tx3ba78bc8924349d18f7af-0053572ccc" "proxy-server 23927" 0.0113</div>
<div><br></div><div>Apr 23 12:00:26 object1 <b>object-server </b>10.41.54.19 - - [23/Apr/2014:03:00:26 +0000] "DELETE /vdb1/861/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615" 409 - "<b>DELETE </b><a href="http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615">http://www.example.com/v2/AUTH_215c458021024a8c87471157a7040644/zoom82/objkey19fbed5e-cb58-4a6d-b4fa-3b5768c7b615</a>" "txe45b336b0ab44aee8a698-0053572ccb" "proxy-server 10071" 0.0003</div>
<div class="gmail_extra"><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Apr 23, 2014 at 5:09 AM, Clay Gerrard <span dir="ltr"><<a href="mailto:clay.gerrard@gmail.com" target="_blank">clay.gerrard@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">409 on DELETE (object?) is a pretty specific error. That should mean that the timestamp assigned to the delete is earlier than the timestamp of the data file.<div>
<br></div><div>Most likely mean that you're getting some time-drift on your proxies (but that assumes multi-node) or maybe that you're reusing names between threads and your object server's see PUT(ts1) PUT(ts3) DELETE(ts2) - but that'd be a pretty tight race...</div>
<div><br></div><div>Should all be logged - try and find a DELETE that went 409 and trace the transaction id.</div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Apr 21, 2014 at 5:54 AM, taurus huang <span dir="ltr"><<a href="mailto:huanggeng.8552@gmail.com" target="_blank">huanggeng.8552@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Please provide the log file： /var/log/swift/swift.log AND /var/log/keystone/keystone.log<br></div><div class="gmail_extra">
<br><br><div class="gmail_quote"><div><div>On Mon, Apr 21, 2014 at 11:55 AM, Sumit Gaur <span dir="ltr"><<a href="mailto:sumitkgaur@gmail.com" target="_blank">sumitkgaur@gmail.com</a>></span> wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div><div dir="ltr"><div class="gmail_extra" style="font-family:arial,sans-serif;font-size:12.727272033691406px">
<div class="gmail_extra">
Hi </div><div class="gmail_extra">I using jclouds lib integrated with Openstack Swift+ keystone combination. Things are working fine except stability test. After 20-30 hours of test jclouds/SWIFT start degrading in TPS and keep going down over the time. </div>
<div class="gmail_extra"><br></div><div class="gmail_extra">1) I am running the (PUT-GET-DEL) cycle in 10 parallel threads.</div><div class="gmail_extra">2) I am getting a lot of 409 and DEL failure for the response too from SWIFT.</div>
<div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra">Can sombody help me what is going wrong here ?</div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks</div><div>
<div><img src="https://mail.google.com/mail/u/0/images/cleardot.gif"></div></div><span><font color="#888888"><span><font color="#888888"><div class="gmail_extra">sumit</div></font></span></font></span></div>
</div>
<br></div></div>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>