Open Stack

On 7/14/2014 5:28 PM, Matt Riedemann wrote:
>
>
> On 7/14/2014 5:18 PM, Ben Nemec wrote:
>> On 07/14/2014 04:21 PM, Matt Riedemann wrote:
>>>
>>>
>>> On 7/14/2014 4:09 PM, Matt Riedemann wrote:
>>>> I opened bug 1341777 [1] against glance but it looks like it's due to
>>>> the default log level for requests.packages.urllib3.connectionpool in
>>>> oslo's log module.
>>>>
>>>> The problem is this warning shows up nearly 420K times in 7 days in
>>>> Tempest runs:
>>>>
>>>> WARNING urllib3.connectionpool [-] HttpConnectionPool is full,
>>>> discarding connection: 127.0.0.1
>>>>
>>>> So either glance is doing something wrong, or that's logging too
>>>> high of
>>>> a level (I think it should be debug in this case).  I'm not really sure
>>>> how to scope this down though, or figure out what is so damn chatty in
>>>> glance-api that is causing this.  It doesn't seem to be causing test
>>>> failures, but the rate at which this is logged in glance-api is
>>>> surprising.
>>>>
>>>> [1] https://bugs.launchpad.net/glance/+bug/1341777
>>>>
>>>
>>> I found this older thread [1] which led to this in oslo [2] but I'm not
>>> really sure how to use it to make the connectionpool logging quieter in
>>> glance, any guidance there?  It looks like in Joe's change to nova for
>>> oslo.messaging he just changed the value directly in the log module in
>>> nova, something I thought was forbidden.
>>>
>>> [1]
>>> http://lists.openstack.org/pipermail/openstack-dev/2014-March/030763.html
>>>
>>> [2] https://review.openstack.org/#/c/94001/
>>>
>>
>> There was a change recently in incubator to address something related,
>> but since it's setting to WARN I don't think it would get rid of this
>> message:
>> https://github.com/openstack/oslo-incubator/commit/3310d8d2d3643da2fc249fdcad8f5000866c4389
>>
>>
>> It looks like Joe's change was a cherry-pick of the incubator change to
>> add oslo.messaging, so discouraged but not forbidden (and apparently
>> during feature freeze, which is understandable).
>>
>> -Ben
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> Yeah it sounds like either a problem in glance because they don't allow
> configuring the max pool size so it defaults to 1, or it's an issue in
> python-swiftclient and is being tracked in a different bug:
>
> https://bugs.launchpad.net/python-swiftclient/+bug/1295812
>

It looks like the issue for the g-api logs was bug 1295812 in 
python-swiftclient, around the time that moved to using python-requests.

I noticed last night that the n-cpu/c-vol logs started spiking with the 
urllib3 connectionpool warning on 7/11 which is when python-glanceclient 
started using requests, so I've changed bug 1341777 to a 
python-glanceclient bug.

-- 

Thanks,

Matt Riedemann