Open Stack

Httplib2 does very little directly with ssl other than a wrap_socket call.
Unless requests has special ssl error handling and retry logic, it will be
exposed to the same set of underlying errors from the ssl library so a
retry that at least catches socket and ssl errors is a good idea.
On May 29, 2014 1:17 PM, "Paul Ward" <wpward at us.ibm.com> wrote:

> Yes, we're still on a code level that uses httplib2.  I noticed that as
> well, but wasn't sure if that would really
> help here as it seems like an ssl thing itself.  But... who knows??  I'm
> not sure how consistently we can
> recreate this, but if we can, I'll try using that patch to use requests
> and see if that helps.
>
>
>
> "Armando M." <armamig at gmail.com> wrote on 05/29/2014 11:52:34 AM:
>
> > From: "Armando M." <armamig at gmail.com>
> > To: "OpenStack Development Mailing List (not for usage questions)"
> > <openstack-dev at lists.openstack.org>,
> > Date: 05/29/2014 11:58 AM
> > Subject: Re: [openstack-dev] [neutron] Supporting retries in
> neutronclient
> >
> > Hi Paul,
> >
> > Just out of curiosity, I am assuming you are using the client that
> > still relies on httplib2. Patch [1] replaced httplib2 with requests,
> > but I believe that a new client that incorporates this change has not
> > yet been published. I wonder if the failures you are referring to
> > manifest themselves with the former http library rather than the
> > latter. Could you clarify?
> >
> > Thanks,
> > Armando
> >
> > [1] - https://review.openstack.org/#/c/89879/
> >
> > On 29 May 2014 17:25, Paul Ward <wpward at us.ibm.com> wrote:
> > > Well, for my specific error, it was an intermittent ssl handshake error
> > > before the request was ever sent to the
> > > neutron-server.  In our case, we saw that 4 out of 5 resize operations
> > > worked, the fifth failed with this ssl
> > > handshake error in neutronclient.
> > >
> > > I certainly think a GET is safe to retry, and I agree with your
> statement
> > > that PUTs and DELETEs probably
> > > are as well.  This still leaves a change in nova needing to be made to
> > > actually a) specify a conf option and
> > > b) pass it to neutronclient where appropriate.
> > >
> > >
> > > Aaron Rosen <aaronorosen at gmail.com> wrote on 05/28/2014 07:38:56 PM:
> > >
> > >> From: Aaron Rosen <aaronorosen at gmail.com>
> > >
> > >
> > >> To: "OpenStack Development Mailing List (not for usage questions)"
> > >> <openstack-dev at lists.openstack.org>,
> > >> Date: 05/28/2014 07:44 PM
> > >
> > >> Subject: Re: [openstack-dev] [neutron] Supporting retries in
> neutronclient
> > >>
> > >> Hi,
> > >>
> > >> I'm curious if other openstack clients implement this type of retry
> > >> thing. I think retrying on GET/DELETES/PUT's should probably be okay.
> > >>
> > >> What types of errors do you see in the neutron-server when it fails
> > >> to respond? I think it would be better to move the retry logic into
> > >> the server around the failures rather than the client (or better yet
> > >> if we fixed the server :)). Most of the times I've seen this type of
> > >> failure is due to deadlock errors caused between (sqlalchemy and
> > >> eventlet *i think*) which cause the client to eventually timeout.
> > >>
> > >> Best,
> > >>
> > >> Aaron
> > >>
> > >
> > >> On Wed, May 28, 2014 at 11:51 AM, Paul Ward <wpward at us.ibm.com>
> wrote:
> > >> Would it be feasible to make the retry logic only apply to read-only
> > >> operations?  This would still require a nova change to specify the
> > >> number of retries, but it'd also prevent invokers from shooting
> > >> themselves in the foot if they call for a write operation.
> > >>
> > >>
> > >>
> > >> Aaron Rosen <aaronorosen at gmail.com> wrote on 05/27/2014 09:40:00 PM:
> > >>
> > >> > From: Aaron Rosen <aaronorosen at gmail.com>
> > >>
> > >> > To: "OpenStack Development Mailing List (not for usage questions)"
> > >> > <openstack-dev at lists.openstack.org>,
> > >> > Date: 05/27/2014 09:44 PM
> > >>
> > >> > Subject: Re: [openstack-dev] [neutron] Supporting retries in
> > >> > neutronclient
> > >> >
> > >> > Hi,
> > >>
> > >> >
> > >> > Is it possible to detect when the ssl handshaking error occurs on
> > >> > the client side (and only retry for that)? If so I think we should
> > >> > do that rather than retrying multiple times. The danger here is
> > >> > mostly for POST operations (as Eugene pointed out) where it's
> > >> > possible for the response to not make it back to the client and for
> > >> > the operation to actually succeed.
> > >> >
> > >> > Having this retry logic nested in the client also prevents things
> > >> > like nova from handling these types of failures individually since
> > >> > this retry logic is happening inside of the client. I think it would
> > >> > be better not to have this internal mechanism in the client and
> > >> > instead make the user of the client implement retry so they are
> > >> > aware of failures.
> > >> >
> > >> > Aaron
> > >> >
> > >>
> > >> > On Tue, May 27, 2014 at 10:48 AM, Paul Ward <wpward at us.ibm.com>
> wrote:
> > >> > Currently, neutronclient is hardcoded to only try a request once in
> > >> > retry_request by virtue of the fact that it uses self.retries as the
> > >> > retry count, and that's initialized to 0 and never changed.  We've
> > >> > seen an issue where we get an ssl handshaking error intermittently
> > >> > (seems like more of an ssl bug) and a retry would probably have
> > >> > worked.  Yet, since neutronclient only tries once and gives up, it
> > >> > fails the entire operation.  Here is the code in question:
> > >> >
> > >> > https://github.com/openstack/python-neutronclient/blob/master/
> > >> > neutronclient/v2_0/client.py#L1296
> > >> >
> > >> > Does anybody know if there's some explicit reason we don't currently
> > >> > allow configuring the number of retries?  If not, I'm inclined to
> > >> > propose a change for just that.
> > >> >
> > >> > _______________________________________________
> > >> > OpenStack-dev mailing list
> > >> > OpenStack-dev at lists.openstack.org
> > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>
> > >> > _______________________________________________
> > >> > OpenStack-dev mailing list
> > >> > OpenStack-dev at lists.openstack.org
> > >> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>
> > >> _______________________________________________
> > >> OpenStack-dev mailing list
> > >> OpenStack-dev at lists.openstack.org
> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > >> _______________________________________________
> > >> OpenStack-dev mailing list
> > >> OpenStack-dev at lists.openstack.org
> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > >
> > > _______________________________________________
> > > OpenStack-dev mailing list
> > > OpenStack-dev at lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140529/6bf317d5/attachment.html>