<div dir="ltr"><div><div>Hello Jonathan,<br></div>Have you seen this discussion on the openstack-dev [1] that discusses the bug 1361357 [2]?<br>I have no idea if it is related to your issue but FWIW a fix backport in the latest Icehouse release introduced performance regression for the metadata agent.<br></div>BR<br>Simon<br><div><br>[1] <a href="http://lists.openstack.org/pipermail/openstack-dev/2014-October/048916.html">http://lists.openstack.org/pipermail/openstack-dev/2014-October/048916.html</a><br>[2] <a href="https://bugs.launchpad.net/cloud-archive/+bug/1361357">https://bugs.launchpad.net/cloud-archive/+bug/1361357</a><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Oct 22, 2014 at 2:33 AM, Jonathan Proulx <span dir="ltr"><<a href="mailto:jon@jonproulx.com" target="_blank">jon@jonproulx.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Ah there's the log many instances of:<br>
<br>
2014-10-21 19:50:15.527 12931 INFO neutron.wsgi [-] 10.10.167.98 - -<br>
[21/Oct/2014 19:50:15] "GET /openstack/2012-08-10 HTTP/1.1" 500 343<br>
120.411705<br>
<br>
2014-10-21 19:50:15.528 12931 ERROR<br>
neutron.agent.metadata.namespace_proxy [-] Unexpected error.<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy Traceback (most recent call<br>
last):<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy File<br>
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py",<br>
line 74, in __cal<br>
l__<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy req.body)<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy File<br>
"/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py",<br>
line 105, in _pro<br>
xy_request<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy<br>
connection_type=UnixDomainHTTPConnection)<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy File<br>
"/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1569, in<br>
request<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy (response, content) =<br>
self._request(conn, authority, uri, request_uri, method, body,<br>
headers, redi<br>
rections, cachekey)<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy File<br>
"/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1316, in<br>
_request<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy (response, content) =<br>
self._conn_request(conn, request_uri, method, body, headers)<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy File<br>
"/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1285, in<br>
_conn_request<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy response =<br>
conn.getresponse()<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy File<br>
"/usr/lib/python2.7/httplib.py", line 1045, in getresponse<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy response.begin()<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy File<br>
"/usr/lib/python2.7/httplib.py", line 409, in begin<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy version, status, reason =<br>
self._read_status()<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy File<br>
"/usr/lib/python2.7/httplib.py", line 373, in _read_status<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy raise BadStatusLine(line)<br>
2014-10-21 19:50:15.528 12931 TRACE<br>
neutron.agent.metadata.namespace_proxy BadStatusLine: ''<br>
2014-10-21 19:50:15.528 12931 TRACE neutron.agent.metadata.namespace_proxy<br>
<div class="HOEnZb"><div class="h5"><br>
On Tue, Oct 21, 2014 at 8:17 PM, Jonathan Proulx <<a href="mailto:jon@jonproulx.com">jon@jonproulx.com</a>> wrote:<br>
> running Icehouse + Neutron ML2/OVS and network names spaces.<br>
><br>
> Was running well unitl recently, most recent change was switching to<br>
> Ceph RBD for ephemeral storage on the hypervisors (and glance). I<br>
> suspect this of being relevant because it makes the instances launch<br>
> much more quickly.<br>
><br>
> I haven't classified the breaking point but launching 64 instances<br>
> deterministically breaks the metadata agent.<br>
><br>
> The service seems to be running on the controller, but is not<br>
> listening in the network namespace. It seems to require restarting<br>
> both the dhcp-agent and the metadata agent to get it to go again.<br>
><br>
> Even in debug mode I get no errors in the logs.<br>
><br>
> Anyone seen this?<br>
><br>
> -Jon<br>
<br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
</div></div></blockquote></div><br></div>