[openstack-dev] [oslo][glance] what to do about tons of http connection pool is full warnings in g-api log?

Matt Riedemann mriedem at linux.vnet.ibm.com
Mon Jul 14 22:28:00 UTC 2014



On 7/14/2014 5:18 PM, Ben Nemec wrote:
> On 07/14/2014 04:21 PM, Matt Riedemann wrote:
>>
>>
>> On 7/14/2014 4:09 PM, Matt Riedemann wrote:
>>> I opened bug 1341777 [1] against glance but it looks like it's due to
>>> the default log level for requests.packages.urllib3.connectionpool in
>>> oslo's log module.
>>>
>>> The problem is this warning shows up nearly 420K times in 7 days in
>>> Tempest runs:
>>>
>>> WARNING urllib3.connectionpool [-] HttpConnectionPool is full,
>>> discarding connection: 127.0.0.1
>>>
>>> So either glance is doing something wrong, or that's logging too high of
>>> a level (I think it should be debug in this case).  I'm not really sure
>>> how to scope this down though, or figure out what is so damn chatty in
>>> glance-api that is causing this.  It doesn't seem to be causing test
>>> failures, but the rate at which this is logged in glance-api is surprising.
>>>
>>> [1] https://bugs.launchpad.net/glance/+bug/1341777
>>>
>>
>> I found this older thread [1] which led to this in oslo [2] but I'm not
>> really sure how to use it to make the connectionpool logging quieter in
>> glance, any guidance there?  It looks like in Joe's change to nova for
>> oslo.messaging he just changed the value directly in the log module in
>> nova, something I thought was forbidden.
>>
>> [1]
>> http://lists.openstack.org/pipermail/openstack-dev/2014-March/030763.html
>> [2] https://review.openstack.org/#/c/94001/
>>
>
> There was a change recently in incubator to address something related,
> but since it's setting to WARN I don't think it would get rid of this
> message:
> https://github.com/openstack/oslo-incubator/commit/3310d8d2d3643da2fc249fdcad8f5000866c4389
>
> It looks like Joe's change was a cherry-pick of the incubator change to
> add oslo.messaging, so discouraged but not forbidden (and apparently
> during feature freeze, which is understandable).
>
> -Ben
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

Yeah it sounds like either a problem in glance because they don't allow 
configuring the max pool size so it defaults to 1, or it's an issue in 
python-swiftclient and is being tracked in a different bug:

https://bugs.launchpad.net/python-swiftclient/+bug/1295812

-- 

Thanks,

Matt Riedemann




More information about the OpenStack-dev mailing list