Open Stack

Hello Jonathan,
Have you seen this discussion on the openstack-dev [1] that discusses the
bug 1361357 [2]?
I have no idea if it is related to your issue but FWIW a fix backport in
the latest Icehouse release introduced performance regression for the
metadata agent.
BR
Simon

[1]
http://lists.openstack.org/pipermail/openstack-dev/2014-October/048916.html
[2] https://bugs.launchpad.net/cloud-archive/+bug/1361357

On Wed, Oct 22, 2014 at 2:33 AM, Jonathan Proulx <jon at jonproulx.com> wrote:

> Ah there's the log many instances of:
>
> 2014-10-21 19:50:15.527 12931 INFO neutron.wsgi [-] 10.10.167.98 - -
> [21/Oct/2014 19:50:15] "GET /openstack/2012-08-10 HTTP/1.1" 500 343
> 120.411705
>
> 2014-10-21 19:50:15.528 12931 ERROR
> neutron.agent.metadata.namespace_proxy [-] Unexpected error.
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy Traceback (most recent call
> last):
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy   File
>
> "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py",
> line 74, in __cal
> l__
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy     req.body)
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy   File
>
> "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py",
> line 105, in _pro
> xy_request
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy
> connection_type=UnixDomainHTTPConnection)
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy   File
> "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1569, in
> request
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy     (response, content) =
> self._request(conn, authority, uri, request_uri, method, body,
> headers, redi
> rections, cachekey)
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy   File
> "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1316, in
> _request
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy     (response, content) =
> self._conn_request(conn, request_uri, method, body, headers)
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy   File
> "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1285, in
> _conn_request
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy     response =
> conn.getresponse()
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy   File
> "/usr/lib/python2.7/httplib.py", line 1045, in getresponse
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy     response.begin()
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy   File
> "/usr/lib/python2.7/httplib.py", line 409, in begin
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy     version, status, reason =
> self._read_status()
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy   File
> "/usr/lib/python2.7/httplib.py", line 373, in _read_status
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy     raise BadStatusLine(line)
> 2014-10-21 19:50:15.528 12931 TRACE
> neutron.agent.metadata.namespace_proxy BadStatusLine: ''
> 2014-10-21 19:50:15.528 12931 TRACE neutron.agent.metadata.namespace_proxy
>
> On Tue, Oct 21, 2014 at 8:17 PM, Jonathan Proulx <jon at jonproulx.com>
> wrote:
> > running Icehouse + Neutron ML2/OVS and network names spaces.
> >
> > Was running well unitl recently, most recent change was switching to
> > Ceph RBD for ephemeral storage on the hypervisors (and glance). I
> > suspect this of being relevant because it makes the instances launch
> > much more quickly.
> >
> > I haven't classified the breaking point but launching 64 instances
> > deterministically breaks the metadata agent.
> >
> > The service seems to be running on the controller, but is not
> > listening in the network namespace.  It seems to require restarting
> > both the dhcp-agent and the metadata agent  to get it to go again.
> >
> > Even in debug mode I get no errors in the logs.
> >
> > Anyone seen this?
> >
> > -Jon
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20141022/5831cbb0/attachment.html>