<p dir="ltr">Httplib2 does very little directly with ssl other than a wrap_socket call. Unless requests has special ssl error handling and retry logic, it will be exposed to the same set of underlying errors from the ssl library so a retry that at least catches socket and ssl errors is a good idea. </p>
<div class="gmail_quote">On May 29, 2014 1:17 PM, "Paul Ward" <<a href="mailto:wpward@us.ibm.com">wpward@us.ibm.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<p><font face="sans-serif">Yes, we're still on a code level that uses httplib2. I noticed that as well, but wasn't sure if that would really</font><br>
<font face="sans-serif">help here as it seems like an ssl thing itself. But... who knows?? I'm not sure how consistently we can</font><br>
<font face="sans-serif">recreate this, but if we can, I'll try using that patch to use requests and see if that helps.<br>
<br>
</font><br>
<br>
<tt><font>"Armando M." <<a href="mailto:armamig@gmail.com" target="_blank">armamig@gmail.com</a>> wrote on 05/29/2014 11:52:34 AM:<br>
<br>
> From: "Armando M." <<a href="mailto:armamig@gmail.com" target="_blank">armamig@gmail.com</a>></font></tt><br>
<tt><font>> To: "OpenStack Development Mailing List (not for usage questions)" <br>
> <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>>, </font></tt><br>
<tt><font>> Date: 05/29/2014 11:58 AM</font></tt><br>
<tt><font>> Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient</font></tt><br>
<tt><font>> <br>
> Hi Paul,<br>
> <br>
> Just out of curiosity, I am assuming you are using the client that<br>
> still relies on httplib2. Patch [1] replaced httplib2 with requests,<br>
> but I believe that a new client that incorporates this change has not<br>
> yet been published. I wonder if the failures you are referring to<br>
> manifest themselves with the former http library rather than the<br>
> latter. Could you clarify?<br>
> <br>
> Thanks,<br>
> Armando<br>
> <br>
> [1] - <a href="https://review.openstack.org/#/c/89879/" target="_blank">https://review.openstack.org/#/c/89879/</a><br>
> <br>
> On 29 May 2014 17:25, Paul Ward <<a href="mailto:wpward@us.ibm.com" target="_blank">wpward@us.ibm.com</a>> wrote:<br>
> > Well, for my specific error, it was an intermittent ssl handshake error<br>
> > before the request was ever sent to the<br>
> > neutron-server. In our case, we saw that 4 out of 5 resize operations<br>
> > worked, the fifth failed with this ssl<br>
> > handshake error in neutronclient.<br>
> ><br>
> > I certainly think a GET is safe to retry, and I agree with your statement<br>
> > that PUTs and DELETEs probably<br>
> > are as well. This still leaves a change in nova needing to be made to<br>
> > actually a) specify a conf option and<br>
> > b) pass it to neutronclient where appropriate.<br>
> ><br>
> ><br>
> > Aaron Rosen <<a href="mailto:aaronorosen@gmail.com" target="_blank">aaronorosen@gmail.com</a>> wrote on 05/28/2014 07:38:56 PM:<br>
> ><br>
> >> From: Aaron Rosen <<a href="mailto:aaronorosen@gmail.com" target="_blank">aaronorosen@gmail.com</a>><br>
> ><br>
> ><br>
> >> To: "OpenStack Development Mailing List (not for usage questions)"<br>
> >> <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>>,<br>
> >> Date: 05/28/2014 07:44 PM<br>
> ><br>
> >> Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient<br>
> >><br>
> >> Hi,<br>
> >><br>
> >> I'm curious if other openstack clients implement this type of retry<br>
> >> thing. I think retrying on GET/DELETES/PUT's should probably be okay.<br>
> >><br>
> >> What types of errors do you see in the neutron-server when it fails<br>
> >> to respond? I think it would be better to move the retry logic into<br>
> >> the server around the failures rather than the client (or better yet<br>
> >> if we fixed the server :)). Most of the times I've seen this type of<br>
> >> failure is due to deadlock errors caused between (sqlalchemy and<br>
> >> eventlet *i think*) which cause the client to eventually timeout.<br>
> >><br>
> >> Best,<br>
> >><br>
> >> Aaron<br>
> >><br>
> ><br>
> >> On Wed, May 28, 2014 at 11:51 AM, Paul Ward <<a href="mailto:wpward@us.ibm.com" target="_blank">wpward@us.ibm.com</a>> wrote:<br>
> >> Would it be feasible to make the retry logic only apply to read-only<br>
> >> operations? This would still require a nova change to specify the<br>
> >> number of retries, but it'd also prevent invokers from shooting<br>
> >> themselves in the foot if they call for a write operation.<br>
> >><br>
> >><br>
> >><br>
> >> Aaron Rosen <<a href="mailto:aaronorosen@gmail.com" target="_blank">aaronorosen@gmail.com</a>> wrote on 05/27/2014 09:40:00 PM:<br>
> >><br>
> >> > From: Aaron Rosen <<a href="mailto:aaronorosen@gmail.com" target="_blank">aaronorosen@gmail.com</a>><br>
> >><br>
> >> > To: "OpenStack Development Mailing List (not for usage questions)"<br>
> >> > <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>>,<br>
> >> > Date: 05/27/2014 09:44 PM<br>
> >><br>
> >> > Subject: Re: [openstack-dev] [neutron] Supporting retries in<br>
> >> > neutronclient<br>
> >> ><br>
> >> > Hi,<br>
> >><br>
> >> ><br>
> >> > Is it possible to detect when the ssl handshaking error occurs on<br>
> >> > the client side (and only retry for that)? If so I think we should<br>
> >> > do that rather than retrying multiple times. The danger here is<br>
> >> > mostly for POST operations (as Eugene pointed out) where it's<br>
> >> > possible for the response to not make it back to the client and for<br>
> >> > the operation to actually succeed.<br>
> >> ><br>
> >> > Having this retry logic nested in the client also prevents things<br>
> >> > like nova from handling these types of failures individually since<br>
> >> > this retry logic is happening inside of the client. I think it would<br>
> >> > be better not to have this internal mechanism in the client and<br>
> >> > instead make the user of the client implement retry so they are<br>
> >> > aware of failures.<br>
> >> ><br>
> >> > Aaron<br>
> >> ><br>
> >><br>
> >> > On Tue, May 27, 2014 at 10:48 AM, Paul Ward <<a href="mailto:wpward@us.ibm.com" target="_blank">wpward@us.ibm.com</a>> wrote:<br>
> >> > Currently, neutronclient is hardcoded to only try a request once in<br>
> >> > retry_request by virtue of the fact that it uses self.retries as the<br>
> >> > retry count, and that's initialized to 0 and never changed. We've<br>
> >> > seen an issue where we get an ssl handshaking error intermittently<br>
> >> > (seems like more of an ssl bug) and a retry would probably have<br>
> >> > worked. Yet, since neutronclient only tries once and gives up, it<br>
> >> > fails the entire operation. Here is the code in question:<br>
> >> ><br>
> >> > <a href="https://github.com/openstack/python-neutronclient/blob/master/" target="_blank">https://github.com/openstack/python-neutronclient/blob/master/</a><br>
> >> > neutronclient/v2_0/client.py#L1296<br>
> >> ><br>
> >> > Does anybody know if there's some explicit reason we don't currently<br>
> >> > allow configuring the number of retries? If not, I'm inclined to<br>
> >> > propose a change for just that.<br>
> >> ><br>
> >> > _______________________________________________<br>
> >> > OpenStack-dev mailing list<br>
> >> > <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> >> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> >><br>
> >> > _______________________________________________<br>
> >> > OpenStack-dev mailing list<br>
> >> > <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> >> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> >><br>
> >> _______________________________________________<br>
> >> OpenStack-dev mailing list<br>
> >> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> >> _______________________________________________<br>
> >> OpenStack-dev mailing list<br>
> >> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > OpenStack-dev mailing list<br>
> > <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> <br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> <br>
</font></tt></p></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div>