Open Stack

mishandling of SSL was the very reason why I brought that change
forward; so I wouldn't rule it out completely ;)

A.

On 29 May 2014 19:15, Paul Ward <wpward at us.ibm.com> wrote:
> Yes, we're still on a code level that uses httplib2.  I noticed that as
> well, but wasn't sure if that would really
> help here as it seems like an ssl thing itself.  But... who knows??  I'm not
> sure how consistently we can
> recreate this, but if we can, I'll try using that patch to use requests and
> see if that helps.
>
>
>
> "Armando M." <armamig at gmail.com> wrote on 05/29/2014 11:52:34 AM:
>
>> From: "Armando M." <armamig at gmail.com>
>
>
>> To: "OpenStack Development Mailing List (not for usage questions)"
>> <openstack-dev at lists.openstack.org>,
>> Date: 05/29/2014 11:58 AM
>
>> Subject: Re: [openstack-dev] [neutron] Supporting retries in neutronclient
>>
>> Hi Paul,
>>
>> Just out of curiosity, I am assuming you are using the client that
>> still relies on httplib2. Patch [1] replaced httplib2 with requests,
>> but I believe that a new client that incorporates this change has not
>> yet been published. I wonder if the failures you are referring to
>> manifest themselves with the former http library rather than the
>> latter. Could you clarify?
>>
>> Thanks,
>> Armando
>>
>> [1] - https://review.openstack.org/#/c/89879/
>>
>> On 29 May 2014 17:25, Paul Ward <wpward at us.ibm.com> wrote:
>> > Well, for my specific error, it was an intermittent ssl handshake error
>> > before the request was ever sent to the
>> > neutron-server.  In our case, we saw that 4 out of 5 resize operations
>> > worked, the fifth failed with this ssl
>> > handshake error in neutronclient.
>> >
>> > I certainly think a GET is safe to retry, and I agree with your
>> > statement
>> > that PUTs and DELETEs probably
>> > are as well.  This still leaves a change in nova needing to be made to
>> > actually a) specify a conf option and
>> > b) pass it to neutronclient where appropriate.
>> >
>> >
>> > Aaron Rosen <aaronorosen at gmail.com> wrote on 05/28/2014 07:38:56 PM:
>> >
>> >> From: Aaron Rosen <aaronorosen at gmail.com>
>> >
>> >
>> >> To: "OpenStack Development Mailing List (not for usage questions)"
>> >> <openstack-dev at lists.openstack.org>,
>> >> Date: 05/28/2014 07:44 PM
>> >
>> >> Subject: Re: [openstack-dev] [neutron] Supporting retries in
>> >> neutronclient
>> >>
>> >> Hi,
>> >>
>> >> I'm curious if other openstack clients implement this type of retry
>> >> thing. I think retrying on GET/DELETES/PUT's should probably be okay.
>> >>
>> >> What types of errors do you see in the neutron-server when it fails
>> >> to respond? I think it would be better to move the retry logic into
>> >> the server around the failures rather than the client (or better yet
>> >> if we fixed the server :)). Most of the times I've seen this type of
>> >> failure is due to deadlock errors caused between (sqlalchemy and
>> >> eventlet *i think*) which cause the client to eventually timeout.
>> >>
>> >> Best,
>> >>
>> >> Aaron
>> >>
>> >
>> >> On Wed, May 28, 2014 at 11:51 AM, Paul Ward <wpward at us.ibm.com> wrote:
>> >> Would it be feasible to make the retry logic only apply to read-only
>> >> operations?  This would still require a nova change to specify the
>> >> number of retries, but it'd also prevent invokers from shooting
>> >> themselves in the foot if they call for a write operation.
>> >>
>> >>
>> >>
>> >> Aaron Rosen <aaronorosen at gmail.com> wrote on 05/27/2014 09:40:00 PM:
>> >>
>> >> > From: Aaron Rosen <aaronorosen at gmail.com>
>> >>
>> >> > To: "OpenStack Development Mailing List (not for usage questions)"
>> >> > <openstack-dev at lists.openstack.org>,
>> >> > Date: 05/27/2014 09:44 PM
>> >>
>> >> > Subject: Re: [openstack-dev] [neutron] Supporting retries in
>> >> > neutronclient
>> >> >
>> >> > Hi,
>> >>
>> >> >
>> >> > Is it possible to detect when the ssl handshaking error occurs on
>> >> > the client side (and only retry for that)? If so I think we should
>> >> > do that rather than retrying multiple times. The danger here is
>> >> > mostly for POST operations (as Eugene pointed out) where it's
>> >> > possible for the response to not make it back to the client and for
>> >> > the operation to actually succeed.
>> >> >
>> >> > Having this retry logic nested in the client also prevents things
>> >> > like nova from handling these types of failures individually since
>> >> > this retry logic is happening inside of the client. I think it would
>> >> > be better not to have this internal mechanism in the client and
>> >> > instead make the user of the client implement retry so they are
>> >> > aware of failures.
>> >> >
>> >> > Aaron
>> >> >
>> >>
>> >> > On Tue, May 27, 2014 at 10:48 AM, Paul Ward <wpward at us.ibm.com>
>> >> > wrote:
>> >> > Currently, neutronclient is hardcoded to only try a request once in
>> >> > retry_request by virtue of the fact that it uses self.retries as the
>> >> > retry count, and that's initialized to 0 and never changed.  We've
>> >> > seen an issue where we get an ssl handshaking error intermittently
>> >> > (seems like more of an ssl bug) and a retry would probably have
>> >> > worked.  Yet, since neutronclient only tries once and gives up, it
>> >> > fails the entire operation.  Here is the code in question:
>> >> >
>> >> > https://github.com/openstack/python-neutronclient/blob/master/
>> >> > neutronclient/v2_0/client.py#L1296
>> >> >
>> >> > Does anybody know if there's some explicit reason we don't currently
>> >> > allow configuring the number of retries?  If not, I'm inclined to
>> >> > propose a change for just that.
>> >> >
>> >> > _______________________________________________
>> >> > OpenStack-dev mailing list
>> >> > OpenStack-dev at lists.openstack.org
>> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >> > _______________________________________________
>> >> > OpenStack-dev mailing list
>> >> > OpenStack-dev at lists.openstack.org
>> >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >> _______________________________________________
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev at lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >> _______________________________________________
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev at lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>