<div dir="ltr">Sure *looks* like an issue with memcache...<div><br></div><div>You might try narrowing down your list of servers in the memcache ring so you limit the places things might go wrong and try work you way back starting with something that works.</div>
<div><br></div><div>Can you get this working:</div><div><br></div><div><div>    clayg@swift:~$ echo $ST_USER</div><div>    test:tester</div><div>    clayg@swift:~$ python -c "from swift.common.memcached import MemcacheRing; print MemcacheRing(['127.0.0.1']).get('AUTH_/user/$ST_USER')"</div>
<div>    AUTH_tkd9a63e1bb2464ba4a618307ce2521bb0</div></div><div><br></div><div>-Clay</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Aug 21, 2013 at 3:08 PM, Snider, Tim <span dir="ltr"><<a href="mailto:Tim.Snider@netapp.com" target="_blank">Tim.Snider@netapp.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">





<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal">I’ve reconfigured  my swift cluster and restarted memcache service on all the swift nodes.  Now I have temp auth tokens expiring immeadiately.
<u></u><u></u></p>
<p class="MsoNormal">I’ve checked memcache server lines in /etc/swift/proxy-service.conf – those seem to be sane. What am I missing?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">root@controller11:~/ssbench-0.2.16# curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' <a href="http://192.168.10.68:8080/auth/v1.0" target="_blank">http://192.168.10.68:8080/auth/v1.0</a><u></u><u></u></span></p>

<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">* About to connect() to 192.168.10.68 port 8080 (#0)<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">*   Trying 192.168.10.68... connected<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> GET /auth/v1.0 HTTP/1.1<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/<a href="http://1.2.3.4" target="_blank">1.2.3.4</a> libidn/1.23 librtmp/2.3<u></u><u></u></span></p>

<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> Host: <a href="http://192.168.10.68:8080" target="_blank">192.168.10.68:8080</a><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> Accept: */*<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> X-Storage-User: test:tester<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> X-Storage-Pass: testing<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> <u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< HTTP/1.1 200 OK<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< X-Storage-Url: <a href="http://192.168.10.88:8080/v1/AUTH_test" target="_blank">http://192.168.10.88:8080/v1/AUTH_test</a><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< X-Storage-Token: AUTH_tk768c95e7330f46efa88d82787a34e9e4<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< X-Auth-Token: AUTH_tk768c95e7330f46efa88d82787a34e9e4<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< X-Trans-Id: tx450df531523b450eaa966b3365250f80<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< Content-Length: 0<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< Date: Wed, 21 Aug 2013 22:00:38 GMT<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< <u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">* Connection #0 to host 192.168.10.68 left intact<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">* Closing connection #0<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">root@controller11:~/ssbench-0.2.16# curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' <a href="http://192.168.10.68:8080/auth/v1.0" target="_blank">http://192.168.10.68:8080/auth/v1.0</a><u></u><u></u></span></p>

<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">* About to connect() to 192.168.10.68 port 8080 (#0)<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">*   Trying 192.168.10.68... connected<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> GET /auth/v1.0 HTTP/1.1<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/<a href="http://1.2.3.4" target="_blank">1.2.3.4</a> libidn/1.23 librtmp/2.3<u></u><u></u></span></p>

<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> Host: <a href="http://192.168.10.68:8080" target="_blank">192.168.10.68:8080</a><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> Accept: */*<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> X-Storage-User: test:tester<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> X-Storage-Pass: testing<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> <u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< HTTP/1.1 200 OK<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< X-Storage-Url: <a href="http://192.168.10.87:8080/v1/AUTH_test" target="_blank">http://192.168.10.87:8080/v1/AUTH_test</a><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< X-Storage-Token: AUTH_tk21ef1ade731e462e91d8d6d3a1d8cbbe<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< X-Auth-Token: AUTH_tk21ef1ade731e462e91d8d6d3a1d8cbbe<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< X-Trans-Id: txe513acba005f42d298eb7c563bb1e6a3<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< Content-Length: 0<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< Date: Wed, 21 Aug 2013 22:00:39 GMT<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< <u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">* Connection #0 to host 192.168.10.68 left intact<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">* Closing connection #0<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">root@controller11:~/ssbench-0.2.16# curl -v -H 'X-Storage-User: test:tester' -H 'X-Storage-Pass: testing' <a href="http://192.168.10.68:8080/auth/v1.0" target="_blank">http://192.168.10.68:8080/auth/v1.0</a><u></u><u></u></span></p>

<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">* About to connect() to 192.168.10.68 port 8080 (#0)<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">*   Trying 192.168.10.68... connected<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> GET /auth/v1.0 HTTP/1.1<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/<a href="http://1.2.3.4" target="_blank">1.2.3.4</a> libidn/1.23 librtmp/2.3<u></u><u></u></span></p>

<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> Host: <a href="http://192.168.10.68:8080" target="_blank">192.168.10.68:8080</a><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> Accept: */*<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> X-Storage-User: test:tester<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> X-Storage-Pass: testing<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">> <u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< HTTP/1.1 200 OK<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< X-Storage-Url: <a href="http://192.168.10.88:8080/v1/AUTH_test" target="_blank">http://192.168.10.88:8080/v1/AUTH_test</a><u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< X-Storage-Token: AUTH_tkf433982fff3d484591826d1a02e54f7c<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< X-Auth-Token: AUTH_tkf433982fff3d484591826d1a02e54f7c<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< X-Trans-Id: txc009e567a213466a9de01ab14004c053<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< Content-Length: 0<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< Date: Wed, 21 Aug 2013 22:00:41 GMT<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">< <u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">* Connection #0 to host 192.168.10.68 left intact<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">* Closing connection #0<u></u><u></u></span></p>
<p class="MsoNormal" style="margin-right:18.75pt;margin-bottom:0in;margin-left:18.75pt;margin-bottom:.0001pt;background:#f0f0f0">
<span style="font-size:10.0pt;font-family:"Courier New"">root@controller11:~/ssbench-0.2.16#
<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Tim<u></u><u></u></span></p>
</div>
</div>

<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div>