

<HTML><BODY><p>Hello.</p><p>I upgraded release OpenStack from Kilo to Mitaka.</p><p>But now I can't start any Instance. At the compute node nova daemons said:</p><p>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher [req-86e4ab2e-229a-438d-87d2-d9ce9f749dbc 79401c3a746740ef826762ca9eaeb207 3029462986ac400ca2c2c6ac4c9af5a7 - - -] Exception during message handling: Timed out waiting for a reply to message ID cb754491dfb14d8793ca6ee05ae420c9<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher incoming.message))<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 110, in wrapped<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher payload)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 359, in decorated_function<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 328, in decorated_function<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in decorated_function<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in decorated_function<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in decorated_function<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2598, in start_instance<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher self._power_on(context, instance)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2566, in _power_on<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher network_info = self.network_api.get_instance_nw_info(context, instance)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 49, in wrapped<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return func(self, context, *args, **kwargs)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 376, in get_instance_nw_info<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher **kwargs)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/base_api.py", line 253, in get_instance_nw_info<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result = self._get_instance_nw_info(context, instance, **kwargs)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 385, in _get_instance_nw_info<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher nw_info = self.network_rpcapi.get_instance_nw_info(context, **args)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 209, in get_instance_nw_info<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher host=host, project_id=project_id)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher retry=self.retry)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher timeout=timeout, retry=retry)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher retry=retry)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result = self._waiter.wait(msg_id, timeout)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher message = self.waiters.get(msg_id, timeout=timeout)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher 'to message ID %s' % msg_id)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher MessagingTimeout: Timed out waiting for a reply to message ID cb754491dfb14d8793ca6ee05ae420c9<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher <br>2016-04-20 12:41:15.432 226509 DEBUG oslo_concurrency.lockutils [req-793ec582-610d-4702-a975-c642bd7fe799 - - - - -] Releasing semaphore "refresh_cache-866156cf-f86b-43c7-9253-68dc1f939c50" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228</p><p>All services has started up (even rabbitmq at controller node). Communications are success between nodes.</p><p>At the controller node I have messages:</p><p>==> rabbitmq/rabbit@controller.log <==</p><p>=INFO REPORT==== 20-Apr-2016::12:40:16 ===<br>accepting AMQP connection <0.3201.0> (10.30.16.129:51314 -> 10.30.17.10:5672)</p><p>==> nova/nova-scheduler.log <==<br>2016-04-20 12:40:16.499 4099 INFO nova.scheduler.host_manager [req-a010da22-3ea1-4019-82ce-5e434ca74692 - - - - -] Successfully synced instances from host 'compute01'.</p><p>That's config for nova at compute node:</p><p>[DEFAULT]<br>dhcpbridge_flagfile=/etc/nova/nova.conf<br>dhcpbridge=/usr/bin/nova-dhcpbridge<br>log-dir=/var/log/nova<br>state_path=/var/lib/nova<br>lock_path=/var/lock/nova<br>force_dhcp_release=True<br>libvirt_use_virtio_for_bridges=True<br>debug=true<br>api_paste_config=/etc/nova/api-paste.ini<br>enabled_apis=osapi_compute,metadata<br>rpc_backend = rabbit<br>auth_strategy = keystone<br>my_ip = 10.30.16.129<br>linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver<br>firewall_driver = nova.virt.firewall.NoopFirewallDriver<br>instance_usage_audit = True<br>instance_usage_audit_period = hour<br>notify_on_state_change = vm_and_task_state<br>notification_driver = messagingv2<br>block_device_allocate_retries=28800<br>block_device_allocate_retries_interval=3<br>allow_resize_to_same_host = True<br>[vnc]<br>enabled = True<br>keymap = en-us<br>vncserver_listen = 0.0.0.0<br>vncserver_proxyclient_address = 10.30.16.129<br>novncproxy_base_url = http://controller:6080/vnc_auto.html<br>[oslo_messaging_rabbit]<br>rabbit_host = controller<br>rabbit_userid = openstack<br>rabbit_password = PASSWORD<br>[keystone_authtoken]<br>auth_uri = http://controller:5000/v2.0<br>identity_uri = http://controller:35357<br>admin_tenant_name = service<br>admin_user = nova<br>admin_password = PASSWORD</p><p>auth_url = http://controller:35357<br>[glance]<br>[oslo_concurrency]<br>lock_path = /var/lib/nova/tmp<br>[neutron]<br>url = http://controller:9696<br>auth_strategy = keystone<br>admin_auth_url = http://controller:35357/v2.0<br>admin_tenant_name = service<br>admin_username = neutron<br>admin_password = PASSWORD<br>[cache]<br>Enabled = true<br>debug_cache_backend = False<br>[libvirt]<br>hw_disk_discard = unmap<br>libvirt_images_rbd_ceph_conf = /etc/ceph/ceph.conf<br>live_migration_flag=VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED,VIR_MIGRATE_NON_SHARED_INC<br>block_migration_flag=VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_LIVE, VIR_MIGRATE_TUNNELLED, VIR_MIGRATE_NON_SHARED_INC<br>disk_cachemodes="network=writeback"</p><p>Neutron's config @compute node:</p><p>[DEFAULT]<br>rpc_backend = rabbit<br>auth_strategy = keystone<br>core_plugin = ml2<br>service_plugins =<br>notify_nova_on_port_status_changes = True<br>notify_nova_on_port_data_changes = True<br>nova_url = http://controller:8774/v2<br>verbose = True<br>debug = false<br>[matchmaker_redis]<br>[matchmaker_ring]<br>[quotas]<br>[agent]<br>root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf<br>[keystone_authtoken]<br>auth_uri = http://controller:5000/v2.0<br>identity_uri = http://controller:35357<br>admin_tenant_name = service<br>admin_user = neutron<br>admin_password = PASSWORD</p><p>[database]<br>connection = mysql://neutron:PASSWORD@controller/neutron<br>[nova]<br>auth_url = http://controller:35357<br>auth_type = password<br>project_domain_id = default<br>user_domain_id = default<br>region_name = RegionOne<br>project_name = service<br>user-name = nova<br>password = PASSWORD<br>[oslo_concurrency]<br>lock_path = $state_path/lock<br>[oslo_policy]<br>[oslo_messaging_amqp]<br>[oslo_messaging_qpid]<br>[oslo_messaging_rabbit]<br>rabbit_host = controller<br>rabbit_userid = openstack<br>rabbit_password = PASSWORD</p><p>nova's config @controller node:</p><p>[DEFAULT]<br>dhcpbridge_flagfile=/etc/nova/nova.conf<br>dhcpbridge=/usr/bin/nova-dhcpbridge<br>log-dir=/var/log/nova<br>state_path=/var/lib/nova<br>lock_path=/var/lock/nova<br>force_dhcp_release=True<br>libvirt_use_virtio_for_bridges=True<br>debug=false<br>api_paste_config=/etc/nova/api-paste.ini<br>enabled_apis=osapi_compute,metadata<br>rpc_backend = rabbit<br>auth_strategy = keystone<br>my_ip = 10.30.17.10<br>linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver<br>firewall_driver = nova.virt.firewall.NoopFirewallDriver<br>block_device_allocate_retries=28800<br>block_device_allocate_retries_interval=3<br>allow_resize_to_same_host = True<br>[api_database]<br>connection = mysql+pymysql://nova:PASSWORD@controller/nova_api?charset=utf8<br>[database]<br>connection = mysql://nova:PASSWORD@controller/nova<br>[oslo_messaging_rabbit]<br>rabbit_host = controller<br>rabbit_userid = openstack<br>rabbit_password = PASSWORD<br>[keystone_authtoken]<br>auth_uri = http://controller:5000/v2.0<br>identity_uri = http://controller:35357<br>admin_tenant_name = service<br>admin_user = nova<br>admin_password = PASSWORD</p><p>[glance]<br>[oslo_concurrency]<br>lock_path = /var/lib/nova/tmp<br>[neutron]<br>url = http://controller:9696<br>auth_strategy = keystone<br>admin_auth_url = http://controller:35357/v2.0<br>admin_tenant_name = service<br>admin_username = neutron<br>admin_password = PASSWORD<br>service_metadata_proxy = True<br>metadata_proxy_shared_secret = SHARED_SECRET<br>[libvirt]<br>hw_disk_discard = unmap<br>live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED</p><p>Neutron's password @controller node:</p><p>[DEFAULT]<br>rpc_backend = rabbit<br>auth_strategy = keystone<br>core_plugin = ml2<br>service_plugins = router<br>notify_nova_on_port_status_changes = True<br>notify_nova_on_port_data_changes = True<br>nova_url = http://controller:8774/v2<br>verbose = True<br>[matchmaker_redis]<br>[matchmaker_ring]<br>[quotas]<br>[agent]<br>root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf<br>[keystone_authtoken]<br>auth_uri = http://controller:5000/v2.0<br>identity_uri = http://controller:35357<br>admin_tenant_name = service<br>admin_user = neutron<br>admin_password = PASSWORD<br>[database]<br>connection = mysql://neutron:PASSWORD@controller/neutron<br>[nova]<br>auth_uri = http://controller:5000/v2.0<br>identity_uri = http://controller:35357<br>admin_tenant_name = service<br>admin_user = nova<br>admin_password = PASSWORD<br>[oslo_concurrency]<br>lock_path = $state_path/lock<br>[oslo_policy]<br>[oslo_messaging_amqp]<br>[oslo_messaging_qpid]<br>[oslo_messaging_rabbit]<br>rabbit_host = controller<br>rabbit_userid = openstack<br>rabbit_password = PASSWORD</p><p>Do you have any idea? Google didn't help (<br><br><br><br>-- <br>Konstantin</p></BODY></HTML>