Open Stack

After our icehouse -> juno upgrade we are noticing sporadic but frequent errors from nova-metadata when trying to serve metadata requests.  The error is the following:

[req-594325c6-44ed-465c-a8e4-bd5a8e5dbdcb None] Failed to get metadata for ip: x.x.x.x 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler Traceback (most recent call last): 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/api/metadata/handler.py, line 150, in _handle_remote_ip_request 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler meta_data = self.get_metadata_by_remote_address(remote_address) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/api/metadata/handler.py, line 82, in get_metadata_by_remote_address 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler data = base.get_metadata_by_address(self.conductor_api, address) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/api/metadata/base.py, line 509, in get_metadata_by_address 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler fixed_ip = network.API().get_fixed_ip_by_address(ctxt, address) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py, line 1010, in get_fixed_ip_by_address 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler uuid_maps = self._get_instance_uuids_by_ip(context, address) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/network/neutronv2/api.py, line 903, in _get_instance_uuids_by_ip 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler data = neutronv2.get_client(context).list_ports(**search_opts) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/nova/network/neutronv2/__init__.py, line 86, in wrapper 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler ret = obj(*args, **kwargs) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 98, in with_params 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler ret = self.function(instance, *args, **kwargs) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 312, in list_ports 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler **_params) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 1334, in list 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler for r in self._pagination(collection, path, **params): 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 1347, in _pagination 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler res = self.get(path, params=params) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 1320, in get 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler headers=headers, params=params) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 1297, in retry_request 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/v2_0/client.py, line 1240, in do_request 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler content_type=self.content_type()) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/client.py, line 180, in do_request 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler **kwargs) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler File /usr/lib/python2.6/site-packages/neutronclient/client.py, line 133, in _cs_request 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler raise exceptions.ConnectionFailed(reason=e) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler ConnectionFailed: Connection to neutron failed: HTTPSConnectionPool(host='openstack.fdqn.com', port=9696): Read timed out. (read timeout=60) 2015-02-19 12:16:45.903 25007 TRACE nova.api.metadata.handler

We have increased the number of neutron workers (40 API and 40 RPC), the Neutron url_timeout interval in nova from 30 to 60 seconds. We are only seeing this issue in production or pre-prod environments are fine.

Is anyone else noticing this or frequent read timeouts when talking to neutron?  Have you found a solution?  What have you tried?

I am thinking of updating a bunch of the oslo (db, messaging, ect ect) packages to the latest versions to see if things get better.
____________________________________________

Kris Lindgren
Senior Linux Systems Engineer
GoDaddy, LLC.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20150219/8a488263/attachment.html>