Open Stack

Hello everybody!

I'm facing a quite confusing problem with my kilo neutron-metadata-agent on
network node.

What's happening is that randomically, my metadata-agent stop working
properly and causes instances to not be launched. The solution is to
restart neutron-metadata-agent service, after a period of time, the error
happens again.

What the logs says:

Accept-Encoding: gzip, deflate^M
Content-Length: 0^M
Content-Type: text/plain^M
Host: 169.254.169.254^M
User-Agent: Python-httplib2/0.8 (gzip)^M
X-Forwarded-For: 10.4.3.211^M
X-Neutron-Router-Id: 55ebf771-8a36-4185-90df-21413638c1fc __call__
/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py:107
2016-03-23 07:00:45.571 31470 DEBUG oslo_concurrency.lockutils [-] Acquired
semaphore
"('neutron.agent.metadata.agent.MetadataProxyHandler._get_router_networks',
'55ebf771-8a36-4185-90df-21413638c1fc')" lock
/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:391
2016-03-23 07:00:45.572 31470 DEBUG oslo_concurrency.lockutils [-]
Releasing semaphore
"('neutron.agent.metadata.agent.MetadataProxyHandler._get_router_networks',
'55ebf771-8a36-4185-90df-21413638c1fc')" lock
/usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:404
2016-03-23 07:00:45.573 31470 DEBUG neutronclient.client [-]
*REQ: curl -i http://controller:35357/tokens
<http://controller:35357/tokens> -X POST -H "User-Agent:
python-neutronclient" -d '{"auth": {"tenantName": null,
"passwordCredentials": {"username": null, "password": null}}}'*
 http_log_req
/usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:130
2016-03-23 07:00:45.579 31470 DEBUG neutronclient.client [-] RESP:404
{'content-length': '93', 'vary': 'X-Auth-Token', 'server': 'Apache/2.4.7
(Ubuntu)', 'date': 'Wed, 23 Mar 2016 10:00:45 GMT', 'content-type':
'application/json', 'x-distribution': 'Ubuntu'} {"error": {"message": "The
resource could not be found.", "code": 404, "title": "Not Found"}}
 http_log_resp
/usr/lib/python2.7/dist-packages/neutronclient/common/utils.py:139
2016-03-23 07:00:45.580 31470 ERROR neutron.agent.metadata.agent [-]
Unexpected error.
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent Traceback
(most recent call last):
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line
109, in __call__
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent
instance_id, tenant_id = self._get_instance_and_tenant_id(req)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line
204, in _get_instance_and_tenant_id
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent     ports
= self._get_ports(remote_address, network_id, router_id)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line
192, in _get_ports
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent
networks = self._get_router_networks(router_id)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 101, in
__call__
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent     return
self._get_from_cache(target_self, *args, **kwargs)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/common/utils.py", line 79, in
_get_from_cache
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent     item =
self.func(target_self, *args, **kwargs)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line
153, in _get_router_networks
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent
internal_ports = self._get_ports_from_server(router_id=router_id)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line
135, in _get_ports_from_server
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent     return
self._get_ports_using_client(filters)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/agent.py", line
177, in _get_ports_using_client
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent     ports
= client.list_ports(**filters)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99,
in with_params
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent     ret =
self.function(instance, *args, **kwargs)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 497,
in list_ports
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent
**_params)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 304,
in list
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent     for r
in self._pagination(collection, path, **params):
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 317,
in _pagination
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent     res =
self.get(path, params=params)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 290,
in get
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent
headers=headers, params=params)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 267,
in retry_request
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent
headers=headers, params=params)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 197,
in do_request
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent
content_type=self.content_type())
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 158, in
do_request
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent
self.authenticate_and_fetch_endpoint_url()
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 123, in
authenticate_and_fetch_endpoint_url
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent
self.authenticate()
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 237, in
authenticate
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent
self._authenticate_keystone()
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent   File
"/usr/lib/python2.7/dist-packages/neutronclient/client.py", line 218, in
_authenticate_keystone
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent     raise
exceptions.Unauthorized(message=resp_body)
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent
Unauthorized: {"error": {"message": "The resource could not be found.",
"code": 404, "title": "Not Found"}}
2016-03-23 07:00:45.580 31470 TRACE neutron.agent.metadata.agent
2016-03-23 07:00:45.584 31470 INFO eventlet.wsgi.server [-]
10.4.3.211,<local> - - [23/Mar/2016 07:00:45] "GET
/latest/meta-data/instance-type HTTP/1.1" 500 229 0.015149
2016-03-23 07:00:45.590 31469 INFO eventlet.wsgi.server [-] (31469)
accepted ''
2016-03-23 07:00:45.592 31469 DEBUG neutron.agent.metadata.agent [-]
Request: GET /latest/meta-data/ami-id HTTP/1.0^M


I don't understand why python-neutronclient gets a token with null values
since other services work normally and only metadata-agent throws this
error.

Thanks!!!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20160323/3230c1e3/attachment.html>