[Openstack] [Nova] Failed to notify nova on events - Specified RPC version cap, 3.0, is too low. Needs to be higher than 3.23.
Marica Antonacci
marica.antonacci at gmail.com
Thu Oct 2 14:23:52 UTC 2014
Dear all,
we are experiencing some problems with our openstack infrastructure after the upgrade from havana to icehouse (on Ubuntu 12.04).
We have successfully upgraded the controller and network nodes (the neutron server is running on the network node) with quite no downtime using the upgrade compatibility flag set to icehouse-compat in nova.conf.
All the existing VMs running on the havana compute nodes were not affected by the upgrade and worked normally. Note that we are using neutron + ml2 plugin with ovs configured for both flat and gre networks. Now, we are encountering problems with the gre network starting new VM on the havana compute nodes whereas the public flat network works fine.
Moreover we are also experiencing some problems as we started to upgrade the compute nodes: we have selected one compute node and executed dist-upgrade using the icehouse repo. Unfortunately, something goes wrong in the VM instatiation on this icehouse compute node and it seems to be related to the external event notification between the neutron server and the nova service /os-server-external-events.
In more details, we get the following:
- on the compute node, a timeout is notified during the instance creation:
WARNING nova.virt.libvirt.driver [req-a1e6e66e-9b18-458a-b31e-0d64eb098e81 769b7fafd6c145ccb8680e398702f14f 57f427029f1a45cc862a0c1321fd5ff6] Timeout waiting for vif plugging callback for instance 65ffc677-0bd4-4ef6-84ef-852fad012fd1
We had to add the following two parameters in the nova.conf file to carry out successfully the VM instantiation:
vif_plugging_is_fatal = False
vif_plugging_timeout = 2
- on the network node we get (/var/log/neutron/server.log):
2014-10-02 15:38:50.232 4559 DEBUG urllib3.connectionpool [-] "POST /v2/15ee65175a6a4548b5422bb276732580/os-server-external-events HTTP/1.1" 500 128 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415
2014-10-02 15:38:50.234 4559 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'status': 'completed', 'tag': u'3d06fab7-5b82-4122-b519-6bed39b78ed9', 'name': 'network-vif-plugged', 'server_uuid': u'65ffc677-0bd4-4ef6-84ef-852fad012fd1'}]
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova Traceback (most recent call last):
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/neutron/notifiers/nova.py", line 222, in send_events
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova batched_events)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/contrib/server_external_events.py", line 39, in create
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova return_raw=True)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 152, in _create
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova _resp, body = self.api.client.post(url, body=body)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 286, in post
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova return self._cs_request(url, 'POST', **kwargs)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova **kwargs)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova resp, body = self.request(url, method, **kwargs)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 236, in request
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova raise exceptions.from_response(resp, body, url, method)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-3d7bf2e7-b7fc-4fd8-bbd3-658072525d05)
2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova
- on the controller node, we get the following error:
2014-10-02 15:38:50.223 9091 AUDIT nova.api.openstack.compute.contrib.server_external_events [req-3d7bf2e7-b7fc-4fd8-bbd3-658072525d05 5a85cf9e9b814c73828cbaff1810e0d9 15ee65175a6a4548b5422bb276732580] Creating event network-vif-plugged:3d06fab7-5b82-4122-b519-6bed39b78ed9 for instance 65ffc677-0bd4-4ef6-84ef-852fad012fd1
2014-10-02 15:38:50.224 9091 ERROR nova.api.openstack [req-3d7bf2e7-b7fc-4fd8-bbd3-658072525d05 5a85cf9e9b814c73828cbaff1810e0d9 15ee65175a6a4548b5422bb276732580] Caught error: Specified RPC version cap, 3.0, is too low. Needs to be higher than 3.23.
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack Traceback (most recent call last):
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return req.get_response(self.application)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack application, catch_exc_info=False)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack app_iter = application(self.environ, start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return resp(environ, start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 582, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return self.app(env, start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return resp(environ, start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return resp(environ, start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack response = self.app(environ, start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return resp(environ, start_response)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return self.func(req, *args, **kwargs)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 917, in __call__
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack content_type, body, accept)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 983, in _process_stack
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1070, in dispatch
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return method(req=request, **action_args)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/server_external_events.py", line 146, in create
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack context, accepted_instances, accepted_events)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3112, in external_instance_event
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack context, instances_by_host[host], events_by_host[host])
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 969, in external_instance_event
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack events=events)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 130, in cast
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack self._check_version_cap(msg.get('version'))
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 115, in _check_version_cap
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack version_cap=self.version_cap)
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack RPCVersionCapError: Specified RPC version cap, 3.0, is too low. Needs to be higher than 3.23.
2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack
2014-10-02 15:38:50.227 9091 INFO nova.api.openstack [req-3d7bf2e7-b7fc-4fd8-bbd3-658072525d05 5a85cf9e9b814c73828cbaff1810e0d9 15ee65175a6a4548b5422bb276732580] http://prisma-cloud.ba.infn.it:8774/v2/15ee65175a6a4548b5422bb276732580/os-server-external-events returned with HTTP 500
What is causing the rpc version mismatch? Here is the list of neutron and nova packages installed on the controller node:
# dpkg -l | grep -E "nova|neutron"
ii nova-ajax-console-proxy 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-api 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-cert 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-common 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-conductor 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-consoleauth 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-doc 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-novncproxy 1:2014.1.2-0ubuntu1.1~cloud0
ii nova-scheduler 1:2014.1.2-0ubuntu1.1~cloud0
ii python-neutronclient 1:2.3.4-0ubuntu1~cloud0
ii python-nova 1:2014.1.2-0ubuntu1.1~cloud0
ii python-novaclient 1:2.17.0-0ubuntu1~cloud0
and on the network node:
# dpkg -l | grep -E "nova|neutron"
ii neutron-common 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-dhcp-agent 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-l3-agent 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-metadata-agent 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-plugin-ml2 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-plugin-openvswitch 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-plugin-openvswitch-agent 1:2014.1.2-0ubuntu1.1~cloud0
ii neutron-server 1:2014.1.2-0ubuntu1.1~cloud0
ii python-neutron 1:2014.1.2-0ubuntu1.1~cloud0
ii python-neutronclient 1:2.3.4-0ubuntu1~cloud0
ii python-novaclient 1:2.17.0-0ubuntu1~cloud0
Any help is appreciated.
Thank you in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20141002/a1aed22f/attachment.html>
More information about the Openstack
mailing list