<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
We finished upgrading to Juno about the time you guys did. Just
checked logs across all environments since the time of the Juno
upgrade and I'm *not* seeing the same errors.<br>
<br>
For comparison here's what we have (mostly out-of-the-box):<br>
<br>
<blockquote>api_workers and rpc_workers = 32<br>
metadata_workers = 32<br>
url_timeout = 30<br>
oslo version = 1.4.1<br>
</blockquote>
Any related errors in the Neutron logs?<br>
Couple seemingly dumb questions related to system limits, but: <br>
<ol>
<li>Could this be a file descriptors limit for the neutron
processes?</li>
<li>Recently we ran into the file descriptors limit in MySQL which
showed up with "sporadic but frequent errors" in Neutron. Under
load is your MySQL fd limit being hit?</li>
<li>Similar limit question for RabbitMQ.</li>
</ol>
<p>Let me know if you want any more comparison info.<br>
</p>
<p>Sean Lynn<br>
Time Warner Cable, Inc.<br>
</p>
<br>
<br>
----------------------<br>
<pre wrap="">Kris G. Lindgren:</pre>
"<br>
<blockquote>
<pre wrap="">After our icehouse -> juno upgrade we are noticing sporadic but frequent errors from nova-metadata when trying to serve metadata requests. The error is the following:
[req-594325c6-44ed-465c-a8e4-bd5a8e5dbdcb None] Failed to get metadata for ip: x.x.x.x 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler Traceback (most recent call last): 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/api/metadata/handler.py, line 150, in _handle_remote_ip_request 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler meta_data = self.get_metadata_by_remote_address(remote_address) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/api/metadata/handler.py, line 82, in get_metadata_by_remote_address 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler data = base.get_metadata_by_address(self.conductor_api, address) </pre>
...<br>
<pre wrap="">We have increased the number of neutron workers (40 API and 40 RPC), the Neutron url_timeout interval in nova from 30 to 60 seconds. We are only seeing this issue in production or pre-prod environments are fine.
Is anyone else noticing this or frequent read timeouts when talking to neutron? Have you found a solution? What have you tried?
I am thinking of updating a bunch of the oslo (db, messaging, ect ect) packages to the latest versions to see if things get better.</pre>
</blockquote>
"<br>
</body>
</html>