<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0);">
After our icehouse -> juno upgrade we are noticing sporadic but frequent errors from nova-metadata when trying to serve metadata requests. The error is the following:</div>
<div style="color: rgb(0, 0, 0);"><br>
</div>
<div><span style="line-height: 20px; white-space: pre-wrap;">[req-594325c6-44ed-465c-a8e4-bd5a8e5dbdcb None] Failed to get metadata for ip: x.x.x.x 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler Traceback (most recent call last): 2015-02-19 12:16:45.903
25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/api/metadata/handler.py, line 150, in _handle_remote_ip_request 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler meta_data = self.get_metadata_by_remote_address(remote_address)
2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/api/metadata/handler.py, line 82, in get_metadata_by_remote_address 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler data = base.get_metadata_by_address(self.conductor_api,
address) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/api/metadata/base.py, line 509, in get_metadata_by_address 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler fixed_ip = network.API().get_fixed_ip_by_address(ctxt,
address) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py, line 1010, in get_fixed_ip_by_address 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler uuid_maps = self._get_instance_uuids_by_ip(context,
address) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py, line 903, in _get_instance_uuids_by_ip 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler data = neutronv2.get_client(context).list_ports(**search_opts)
2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/network/neutronv2/__init__.py, line 86, in wrapper 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler ret = obj(*args, **kwargs) 2015-02-19
12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 98, in with_params 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler ret = self.function(instance, *args, **kwargs) 2015-02-19
12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 312, in list_ports 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler **_params) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler
File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 1334, in list 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler for r in self._pagination(collection, path, **params): 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler
File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 1347, in _pagination 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler res = self.get(path, params=params) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File
/usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 1320, in get 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler headers=headers, params=params) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py,
line 1297, in retry_request 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 1240, in do_request 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler content_type=self.content_type())
2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/client.py, line 180, in do_request 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler **kwargs) 2015-02-19 12:16:45.903 25007 TRACE
nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/client.py, line 133, in _cs_request 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler raise exceptions.ConnectionFailed(reason=e) 2015-02-19 12:16:45.903 25007 TRACE
nova.api.metadata.handler ConnectionFailed: Connection to neutron failed: HTTPSConnectionPool(host='openstack.fdqn.com', port=9696): Read timed out. (read timeout=60) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler</span></div>
<div><span style="line-height: 20px; white-space: pre-wrap;"><br>
</span></div>
<div><span style="line-height: 20px; white-space: pre-wrap;">We have increased the number of neutron workers (40 API and 40 RPC), the Neutron url_timeout interval in nova from 30 to 60 seconds. We are only seeing this issue in production or pre-prod
</span><span style="line-height: 20px; white-space: pre-wrap;">environments</span><span style="line-height: 20px; white-space: pre-wrap;"> are fine.
</span></div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0);">
<div><br>
</div>
<div>Is anyone else noticing this or frequent read timeouts when talking to neutron? Have you found a solution? What have you tried?</div>
<div><br>
</div>
<div>I am thinking of updating a bunch of the oslo (db, messaging, ect ect) packages to the latest versions to see if things get better.</div>
<div>____________________________________________</div>
<div> </div>
<div>Kris Lindgren</div>
<div>Senior Linux Systems Engineer</div>
<div>GoDaddy, LLC.</div>
</div>
</body>
</html>