[openstack-dev] [oslo][glance] what to do about tons of http connection pool is full warnings in g-api log?

Ben Nemec openstack at nemebean.com
Mon Jul 14 22:18:32 UTC 2014


On 07/14/2014 04:21 PM, Matt Riedemann wrote:
> 
> 
> On 7/14/2014 4:09 PM, Matt Riedemann wrote:
>> I opened bug 1341777 [1] against glance but it looks like it's due to
>> the default log level for requests.packages.urllib3.connectionpool in
>> oslo's log module.
>>
>> The problem is this warning shows up nearly 420K times in 7 days in
>> Tempest runs:
>>
>> WARNING urllib3.connectionpool [-] HttpConnectionPool is full,
>> discarding connection: 127.0.0.1
>>
>> So either glance is doing something wrong, or that's logging too high of
>> a level (I think it should be debug in this case).  I'm not really sure
>> how to scope this down though, or figure out what is so damn chatty in
>> glance-api that is causing this.  It doesn't seem to be causing test
>> failures, but the rate at which this is logged in glance-api is surprising.
>>
>> [1] https://bugs.launchpad.net/glance/+bug/1341777
>>
> 
> I found this older thread [1] which led to this in oslo [2] but I'm not 
> really sure how to use it to make the connectionpool logging quieter in 
> glance, any guidance there?  It looks like in Joe's change to nova for 
> oslo.messaging he just changed the value directly in the log module in 
> nova, something I thought was forbidden.
> 
> [1] 
> http://lists.openstack.org/pipermail/openstack-dev/2014-March/030763.html
> [2] https://review.openstack.org/#/c/94001/
> 

There was a change recently in incubator to address something related,
but since it's setting to WARN I don't think it would get rid of this
message:
https://github.com/openstack/oslo-incubator/commit/3310d8d2d3643da2fc249fdcad8f5000866c4389

It looks like Joe's change was a cherry-pick of the incubator change to
add oslo.messaging, so discouraged but not forbidden (and apparently
during feature freeze, which is understandable).

-Ben



More information about the OpenStack-dev mailing list