Open Stack

On 02/20/2015 03:10 PM, Sean Lynn wrote:
> Sorry, that wasn't exactly clear and my memory was a bit foggy, Jay.
>
> max_connections possibly paired with open_files_limit
>
> https://dev.mysql.com/doc/refman/5.6/en/too-many-connections.html

No worries :) I'd be surprised if you were hitting open_files_limit, as 
that typically is seen when the schemas have hundreds or thousands of 
tables in them. More likely you hit issues with max_connections, which, 
for some reason I cannot fathom, is set to 151 by default in MySQL. :(

I recommend using 2000 or more as the value for max_connections for any 
database server used in production deployments.

Best,
-jay

> On 02/20/15 10:20, Jay Pipes wrote:
>> On 02/20/2015 10:39 AM, Sean Lynn wrote:
>>> We finished upgrading to Juno about the time you guys did.  Just checked
>>> logs across all environments since the time of the Juno upgrade and I'm
>>> *not* seeing the same errors.
>>>
>>> For comparison here's what we have (mostly out-of-the-box):
>>>
>>>     api_workers and rpc_workers = 32
>>>     metadata_workers = 32
>>>     url_timeout = 30
>>>     oslo version = 1.4.1
>>>
>>> Any related errors in the Neutron logs?
>>> Couple seemingly dumb questions related to system limits, but:
>>>
>>>  1. Could this be a file descriptors limit for the neutron processes?
>>>  2. Recently we ran into the file descriptors limit in MySQL which
>>>     showed up with "sporadic but frequent errors" in Neutron. Under
>>>     load is your MySQL fd limit being hit?
>>
>> I'm not familiar with file descriptor limit issues with MySQL. Do you
>> mean max_connections issues?
>>
>> Best,
>> -jay
>>
>>>  3. Similar limit question for RabbitMQ.
>>>
>>> Let me know if you want any more comparison info.
>>>
>>> Sean Lynn
>>> Time Warner Cable, Inc.
>>>
>>>
>>>
>>> ----------------------
>>>
>>> Kris G. Lindgren:
>>>
>>> "
>>>
>>>     After our icehouse -> juno upgrade we are noticing sporadic but
>>> frequent errors from nova-metadata when trying to serve metadata
>>> requests.  The error is the following:
>>>
>>>     [req-594325c6-44ed-465c-a8e4-bd5a8e5dbdcb None] Failed to get
>>> metadata for ip: x.x.x.x 2015-02-19 12:16:45.903 25007 TRACE
>>> nova.api.metadata.handler Traceback (most recent call last):
>>> 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File
>>> /usr/lib/python2.6/site-packages/nova/api/metadata/handler.py, line
>>> 150, in _handle_remote_ip_request 2015-02-19 12:16:45.903 25007 TRACE
>>> nova.api.metadata.handler meta_data =
>>> self.get_metadata_by_remote_address(remote_address) 2015-02-19
>>> 12:16:45.903 25007 TRACE nova.api.metadata.handler File
>>> /usr/lib/python2.6/site-packages/nova/api/metadata/handler.py, line
>>> 82, in get_metadata_by_remote_address 2015-02-19 12:16:45.903 25007
>>> TRACE nova.api.metadata.handler data =
>>> base.get_metadata_by_address(self.conductor_api, address)
>>>
>>>     ...
>>>
>>>     We have increased the number of neutron workers (40 API and 40
>>> RPC), the Neutron url_timeout interval in nova from 30 to 60 seconds.
>>> We are only seeing this issue in production or pre-prod environments
>>> are fine.
>>>
>>>     Is anyone else noticing this or frequent read timeouts when
>>> talking to neutron?  Have you found a solution?  What have you tried?
>>>
>>>     I am thinking of updating a bunch of the oslo (db, messaging, ect
>>> ect) packages to the latest versions to see if things get better.
>>>
>>> "
>>>
>>>
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators