Open Stack

Thanks Simon,

That seems to be it, may find more as I test it but first pass adding
the cache_url got by 64 instance test case to laumch without crashing
the metadata service.

-Jon

On Wed, Oct 22, 2014 at 3:25 AM, Simon Pasquier <spasquier at mirantis.com> wrote:
> Hello Jonathan,
> Have you seen this discussion on the openstack-dev [1] that discusses the
> bug 1361357 [2]?
> I have no idea if it is related to your issue but FWIW a fix backport in the
> latest Icehouse release introduced performance regression for the metadata
> agent.
> BR
> Simon
>
> [1]
> http://lists.openstack.org/pipermail/openstack-dev/2014-October/048916.html
> [2] https://bugs.launchpad.net/cloud-archive/+bug/1361357
>
> On Wed, Oct 22, 2014 at 2:33 AM, Jonathan Proulx <jon at jonproulx.com> wrote:
>>
>> Ah there's the log many instances of:
>>
>> 2014-10-21 19:50:15.527 12931 INFO neutron.wsgi [-] 10.10.167.98 - -
>> [21/Oct/2014 19:50:15] "GET /openstack/2012-08-10 HTTP/1.1" 500 343
>> 120.411705
>>
>> 2014-10-21 19:50:15.528 12931 ERROR
>> neutron.agent.metadata.namespace_proxy [-] Unexpected error.
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy Traceback (most recent call
>> last):
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy   File
>>
>> "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py",
>> line 74, in __cal
>> l__
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy     req.body)
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy   File
>>
>> "/usr/lib/python2.7/dist-packages/neutron/agent/metadata/namespace_proxy.py",
>> line 105, in _pro
>> xy_request
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy
>> connection_type=UnixDomainHTTPConnection)
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy   File
>> "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1569, in
>> request
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy     (response, content) =
>> self._request(conn, authority, uri, request_uri, method, body,
>> headers, redi
>> rections, cachekey)
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy   File
>> "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1316, in
>> _request
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy     (response, content) =
>> self._conn_request(conn, request_uri, method, body, headers)
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy   File
>> "/usr/lib/python2.7/dist-packages/httplib2/__init__.py", line 1285, in
>> _conn_request
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy     response =
>> conn.getresponse()
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy   File
>> "/usr/lib/python2.7/httplib.py", line 1045, in getresponse
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy     response.begin()
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy   File
>> "/usr/lib/python2.7/httplib.py", line 409, in begin
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy     version, status, reason =
>> self._read_status()
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy   File
>> "/usr/lib/python2.7/httplib.py", line 373, in _read_status
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy     raise BadStatusLine(line)
>> 2014-10-21 19:50:15.528 12931 TRACE
>> neutron.agent.metadata.namespace_proxy BadStatusLine: ''
>> 2014-10-21 19:50:15.528 12931 TRACE neutron.agent.metadata.namespace_proxy
>>
>> On Tue, Oct 21, 2014 at 8:17 PM, Jonathan Proulx <jon at jonproulx.com>
>> wrote:
>> > running Icehouse + Neutron ML2/OVS and network names spaces.
>> >
>> > Was running well unitl recently, most recent change was switching to
>> > Ceph RBD for ephemeral storage on the hypervisors (and glance). I
>> > suspect this of being relevant because it makes the instances launch
>> > much more quickly.
>> >
>> > I haven't classified the breaking point but launching 64 instances
>> > deterministically breaks the metadata agent.
>> >
>> > The service seems to be running on the controller, but is not
>> > listening in the network namespace.  It seems to require restarting
>> > both the dhcp-agent and the metadata agent  to get it to go again.
>> >
>> > Even in debug mode I get no errors in the logs.
>> >
>> > Anyone seen this?
>> >
>> > -Jon
>>
>> _______________________________________________
>> OpenStack-operators mailing list
>> OpenStack-operators at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>