Open Stack

Excerpts from Ian Cordasco's message of 2014-09-17 12:42:57 -0700:
> On 9/17/14, 1:46 PM, "Clint Byrum" <clint at fewbar.com> wrote:
> 
> >Excerpts from Davanum Srinivas's message of 2014-09-17 10:15:29 -0700:
> >> I was trying request-ifying oslo.vmware and ran into this as well:
> >> https://review.openstack.org/#/c/121956/
> >> 
> >> And we don't seem to have urllib3 in global-requirements either.
> >> Should we do that first?
> >
> >Honestly, after reading this:
> >
> >https://github.com/kennethreitz/requests/pull/1812
> >
> >I think we might want to consider requests a poor option. Its author
> >clearly doesn't understand the role a _library_ plays in software
> >development and considers requests an application, not a library.
> 
> Yes that is Kenneth’s opinion. That is not the opinion of the core
> developers though. We see it as a library but this is something we aren’t
> going to currently change any time soon.
> 

Good to know but troubling to hear that there's no change in sight.

> >For instance, why is requests exposing internal implementation details
> >at all?
> 
> Where exactly are we exposing internal implementation details? A normal
> user (even advanced users) can use requests without ever digging into
> requests.packages. What implementation details are we exposing and where?
> 
> >It should be wrapping any exceptions or objects to avoid
> >forcing users to make this choice at all.
> 
> We do. Occasionally (like in 2.4.0) urllib3 adds an exception that we
> missed notice of and it slips through. We released 2.4.1 a couple days
> later with the fix for that. Pretty much every error we’ve seen or know
> about is caught and rewrapped as a requests exception. I’m not sure what
> you’re arguing here, unless of course you have not used requests.
> 

I had seen handling of urllib3 exceptions in some requests code and was
seeing a few issues about that. I understand now that it is not
intentional, but a side effect of misunderstanding.

> That aside, I’ve been mulling over how effectively the clients use
> requests. I haven’t investigated all of them, but many seem to reach into
> implementation details on their own. If I remember nova client has
> something it has commented as “connection pooling” while requests and
> urllib3 do that automatically. I haven’t started to investigate exactly
> why they do this. Likewise, glance client has custom certificate
> verification in glanceclient.common.https. Why? I’m not exactly certain
> yet. It seems for the most part from what little I’ve seen that requests
> is too high-level a library for OpenStack’s needs at best, and actively
> obscures details OpenStack developers need (or don’t realize requests
> provides in most cases).
> 

Indeed, it sounds like there has been some misuse of requests that led
to the situation.

> Circling back to the issue of vendoring though: it’s a conscious decision
> to do this, and in the last two years there have been 2 CVEs reported for
> requests. There have been none for urllib3 and none for chardet. (Frankly
> I don’t think either urllib3 or chardet have had any CVEs reported against
> them, but let’s ignore that for now.) While security is typically the
> chief concern with vendoring, none of the libraries we use have had
> security issues rendering it a moot point in my opinion. The benefits of
> vendoring for us as a team have been numerous and we will likely continue
> to do it until it stops benefiting us and our users.
> 

On a micro scale, you are absolutely correct. The point is not that
there is a tactical reason for urllib3 to be de-vendored, but rather
that there is a strategic reason to not use vendored libraries. When
updates are necessary, they must be complete, and rapid. Please refer
to the zlib travesty to see why everyone, not just distribution vendors,
should be extremely concerned about embedded libraries.