<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">Dear all,<br><br>we are experiencing some problems with our openstack infrastructure after the upgrade from havana to icehouse (on Ubuntu 12.04).<br><br>We have successfully upgraded the controller and network nodes (the neutron server is running on the network node) with quite no downtime using the upgrade compatibility flag set to icehouse-compat in nova.conf.<br><br>All the existing VMs running on the <b>havana compute nodes</b> were not affected by the upgrade and worked normally. Note that we are using neutron + ml2 plugin with ovs configured for both flat and gre networks. Now, we are encountering problems with the gre network starting new VM on the <b>havana compute nodes</b> whereas the public flat network works fine.<br><br>Moreover we are also experiencing some problems as we started to upgrade the compute nodes: we have selected one compute node and executed dist-upgrade using the icehouse repo. Unfortunately, something goes wrong in the VM instatiation on this <b>icehouse</b> compute node and it seems to be related to the external event notification between the neutron server and the nova service /os-server-external-events. <div><br></div><div>In more details, we get the following:<br><b>- on the compute node, a timeout is notified during the instance creation:</b><br><div><br></div><div>WARNING nova.virt.libvirt.driver [req-a1e6e66e-9b18-458a-b31e-0d64eb098e81 769b7fafd6c145ccb8680e398702f14f 57f427029f1a45cc862a0c1321fd5ff6] Timeout waiting for vif plugging callback for instance 65ffc677-0bd4-4ef6-84ef-852fad012fd1 </div><div><br></div>We had to add the following two parameters in the nova.conf file to carry out successfully the VM instantiation:<br><br><div>vif_plugging_is_fatal = False</div><div>vif_plugging_timeout = 2</div><br class="Apple-interchange-newline"><div><b>- on the network node we get (/var/log/neutron/server.log):</b><br></div><div>2014-10-02 15:38:50.232 4559 DEBUG urllib3.connectionpool [-] "POST /v2/15ee65175a6a4548b5422bb276732580/os-server-external-events HTTP/1.1" 500 128 _make_request /usr/lib/python2.7/dist-packages/urllib3/connectionpool.py:415</div><div>2014-10-02 15:38:50.234 4559 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'status': 'completed', 'tag': u'3d06fab7-5b82-4122-b519-6bed39b78ed9', 'name': 'network-vif-plugged', 'server_uuid': u'65ffc677-0bd4-4ef6-84ef-852fad012fd1'}]</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova Traceback (most recent call last):</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/neutron/notifiers/nova.py", line 222, in send_events</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova batched_events)</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/v1_1/contrib/server_external_events.py", line 39, in create</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova return_raw=True)</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 152, in _create</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova _resp, body = self.api.client.post(url, body=body)</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 286, in post</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova return self._cs_request(url, 'POST', **kwargs)</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 260, in _cs_request</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova **kwargs)</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 242, in _time_request</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova resp, body = self.request(url, method, **kwargs)</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 236, in request</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova raise exceptions.from_response(resp, body, url, method)</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-3d7bf2e7-b7fc-4fd8-bbd3-658072525d05)</div><div>2014-10-02 15:38:50.234 4559 TRACE neutron.notifiers.nova</div><div><br></div><div><b>- on the controller node, we get the following error:</b><br></div><div>2014-10-02 15:38:50.223 9091 AUDIT nova.api.openstack.compute.contrib.server_external_events [req-3d7bf2e7-b7fc-4fd8-bbd3-658072525d05 5a85cf9e9b814c73828cbaff1810e0d9 15ee65175a6a4548b5422bb276732580] Creating event network-vif-plugged:3d06fab7-5b82-4122-b519-6bed39b78ed9 for instance 65ffc677-0bd4-4ef6-84ef-852fad012fd1</div><div>2014-10-02 15:38:50.224 9091 ERROR nova.api.openstack [req-3d7bf2e7-b7fc-4fd8-bbd3-658072525d05 5a85cf9e9b814c73828cbaff1810e0d9 15ee65175a6a4548b5422bb276732580] Caught error: Specified RPC version cap, 3.0, is too low. Needs to be higher than 3.23.</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack Traceback (most recent call last):</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line 125, in __call__</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return req.get_response(self.application)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in send</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack application, catch_exc_info=False)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in call_application</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack app_iter = application(self.environ, start_response)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return resp(environ, start_response)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py", line 582, in __call__</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return self.app(env, start_response)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return resp(environ, start_response)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return resp(environ, start_response)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in __call__</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack response = self.app(environ, start_response)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return resp(environ, start_response)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return self.func(req, *args, **kwargs)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 917, in __call__</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack content_type, body, accept)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 983, in _process_stack</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line 1070, in dispatch</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack return method(req=request, **action_args)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/contrib/server_external_events.py", line 146, in create</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack context, accepted_instances, accepted_events)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 3112, in external_instance_event</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack context, instances_by_host[host], events_by_host[host])</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/nova/compute/rpcapi.py", line 969, in external_instance_event</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack events=events)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 130, in cast</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack self._check_version_cap(msg.get('version'))</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack File "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line 115, in _check_version_cap</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack version_cap=self.version_cap)</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack RPCVersionCapError: Specified RPC version cap, 3.0, is too low. Needs to be higher than 3.23.</div><div>2014-10-02 15:38:50.224 9091 TRACE nova.api.openstack</div><div>2014-10-02 15:38:50.227 9091 INFO nova.api.openstack [req-3d7bf2e7-b7fc-4fd8-bbd3-658072525d05 5a85cf9e9b814c73828cbaff1810e0d9 15ee65175a6a4548b5422bb276732580] <a href="http://prisma-cloud.ba.infn.it:8774/v2/15ee65175a6a4548b5422bb276732580/os-server-external-events">http://prisma-cloud.ba.infn.it:8774/v2/15ee65175a6a4548b5422bb276732580/os-server-external-events</a> returned with HTTP 500</div><br class="Apple-interchange-newline">What is causing the rpc version mismatch? Here is the list of neutron and nova packages installed on the controller node:<br><br><div># dpkg -l | grep -E "nova|neutron"</div><div>ii nova-ajax-console-proxy 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii nova-api 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii nova-cert 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii nova-common 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii nova-conductor 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii nova-consoleauth 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii nova-doc 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii nova-novncproxy 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii nova-scheduler 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii python-neutronclient 1:2.3.4-0ubuntu1~cloud0 </div><div>ii python-nova 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii python-novaclient 1:2.17.0-0ubuntu1~cloud0 </div><br class="Apple-interchange-newline">and on the network node:<br><br><div># dpkg -l | grep -E "nova|neutron"</div><div>ii neutron-common 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii neutron-dhcp-agent 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii neutron-l3-agent 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii neutron-metadata-agent 1:2014.1.2-0ubuntu1.1~cloud0</div><div>ii neutron-plugin-ml2 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii neutron-plugin-openvswitch 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii neutron-plugin-openvswitch-agent 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii neutron-server 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii python-neutron 1:2014.1.2-0ubuntu1.1~cloud0 </div><div>ii python-neutronclient 1:2.3.4-0ubuntu1~cloud0 </div><div>ii python-novaclient 1:2.17.0-0ubuntu1~cloud0 </div><br class="Apple-interchange-newline">Any help is appreciated. </div><div>Thank you in advance.</div></body></html>