Open Stack

Ah there's the log many instances of:

2014-10-21 19:50:15.527 12931 INFO neutron.wsgi [-] 10.10.167.98 - -
[21/Oct/2014 19:50:15] "GET /openstack/2012-08-10 HTTP/1.1" 500 343
120.411705

2014-10-21 19:50:15.528 12931 ERROR
neutron.agent.metadata.namespace_proxy [-] Unexpected error.
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy Traceback (most recent call
last):
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py",
line 74, in __cal
l__
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy     req.body)
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy   File
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py",
line 105, in _pro
xy_request
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy
connection_type=UnixDomainHTTPConnection)
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy   File
"/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1569, in
request
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy     (response, content) =
self._request(conn, authority, uri, request_uri, method, body,
headers, redi
rections, cachekey)
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy   File
"/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1316, in
_request
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy     (response, content) =
self._conn_request(conn, request_uri, method, body, headers)
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy   File
"/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1285, in
_conn_request
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy     response =
conn.getresponse()
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy   File
"/usr/lib/python2.7/httplib.py", line 1045, in getresponse
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy     response.begin()
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy   File
"/usr/lib/python2.7/httplib.py", line 409, in begin
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy     version, status, reason =
self._read_status()
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy   File
"/usr/lib/python2.7/httplib.py", line 373, in _read_status
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy     raise BadStatusLine(line)
2014-10-21 19:50:15.528 12931 TRACE
neutron.agent.metadata.namespace_proxy BadStatusLine: ''
2014-10-21 19:50:15.528 12931 TRACE neutron.agent.metadata.namespace_proxy

On Tue, Oct 21, 2014 at 8:17 PM, Jonathan Proulx <jon at jonproulx.com> wrote:
> running Icehouse + Neutron ML2/OVS and network names spaces.
>
> Was running well unitl recently, most recent change was switching to
> Ceph RBD for ephemeral storage on the hypervisors (and glance). I
> suspect this of being relevant because it makes the instances launch
> much more quickly.
>
> I haven't classified the breaking point but launching 64 instances
> deterministically breaks the metadata agent.
>
> The service seems to be running on the controller, but is not
> listening in the network namespace.  It seems to require restarting
> both the dhcp-agent and the metadata agent  to get it to go again.
>
> Even in debug mode I get no errors in the logs.
>
> Anyone seen this?
>
> -Jon