<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,102,0)">Hello KK, </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,102,0)"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,102,0)">I see that your config files are up dated, for example lock_path = /var/lib/nova/tmp and auth_password to auth_type. </div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;font-size:small;color:rgb(0,102,0)">Are the endpoint versions correct? Just taking a wild guess. </div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><span style="color:rgb(0,102,0)"><font size="2"><br><font face="arial, helvetica, sans-serif">Best regards,<br></font></font></span><div><span style="color:rgb(0,102,0)"><font size="2" face="arial, helvetica, sans-serif">
Venu</font></span></div><div><br></div><div><div style="color:rgb(51,51,51);font-size:12px;line-height:1.6em"><dl style="font-family:OpenSans-Light,'Open Sans Light','Open Sans','Helvetica Neue',Arial,sans-serif"><div><a href="http://www.thoughtworks.com/" style="font-family:'Open Sans',Helvetica,Arial,sans-serif;font-size:medium;line-height:25.600000381469727px;margin-top:1.5em;display:inline!important" target="_blank"><img src="http://gentle-reef-2837.herokuapp.com/images/thoughtworks_logo_green.png" width="200" height="31" alt="ThoughtWorks" title="ThoughtWorks" style="border-style:none;display:block"></a></div></dl></div></div><div>---<br><div><i><font color="#38761d"><span style="font-weight:normal"><font size="1">“Excellence is never an accident. It is always the result of high intention, sincere effort, and intelligent execution<font size="1">...</font>.</font></span><font size="1"><font size="1"> </font><span style="font-weight:normal">― Aristotle</span></font></font></i>
</div></div><div><div><br></div></div></div></div></div>
<br><div class="gmail_quote">On Wed, Apr 20, 2016 at 2:57 PM, K K <span dir="ltr"><<a href="mailto:nnex@mail.ru" target="_blank">nnex@mail.ru</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><p>Hello.</p><p>I upgraded release OpenStack from Kilo to Mitaka.</p><p>But now I can't start any Instance. At the compute node nova daemons said:</p><p>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher [req-86e4ab2e-229a-438d-87d2-d9ce9f749dbc 79401c3a746740ef826762ca9eaeb207 3029462986ac400ca2c2c6ac4c9af5a7 - - -] Exception during message handling: Timed out waiting for a reply to message ID cb754491dfb14d8793ca6ee05ae420c9<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher incoming.message))<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 110, in wrapped<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher payload)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 359, in decorated_function<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 328, in decorated_function<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in decorated_function<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in decorated_function<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in decorated_function<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2598, in start_instance<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher self._power_on(context, instance)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2566, in _power_on<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher network_info = self.network_api.get_instance_nw_info(context, instance)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 49, in wrapped<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return func(self, context, *args, **kwargs)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 376, in get_instance_nw_info<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher **kwargs)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/base_api.py", line 253, in get_instance_nw_info<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result = self._get_instance_nw_info(context, instance, **kwargs)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 385, in _get_instance_nw_info<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher nw_info = self.network_rpcapi.get_instance_nw_info(context, **args)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 209, in get_instance_nw_info<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher host=host, project_id=project_id)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher retry=self.retry)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher timeout=timeout, retry=retry)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher retry=retry)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result = self._waiter.wait(msg_id, timeout)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher message = self.waiters.get(msg_id, timeout=timeout)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher 'to message ID %s' % msg_id)<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher MessagingTimeout: Timed out waiting for a reply to message ID cb754491dfb14d8793ca6ee05ae420c9<br>2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher <br>2016-04-20 12:41:15.432 226509 DEBUG oslo_concurrency.lockutils [req-793ec582-610d-4702-a975-c642bd7fe799 - - - - -] Releasing semaphore "refresh_cache-866156cf-f86b-43c7-9253-68dc1f939c50" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228</p><p>All services has started up (even rabbitmq at controller node). Communications are success between nodes.</p><p>At the controller node I have messages:</p><p>==> rabbitmq/rabbit@controller.log <==</p><p>=INFO REPORT==== 20-Apr-2016::12:40:16 ===<br>accepting AMQP connection <0.3201.0> (<a href="http://10.30.16.129:51314" target="_blank">10.30.16.129:51314</a> -> <a href="http://10.30.17.10:5672" target="_blank">10.30.17.10:5672</a>)</p><p>==> nova/nova-scheduler.log <==<br>2016-04-20 12:40:16.499 4099 INFO nova.scheduler.host_manager [req-a010da22-3ea1-4019-82ce-5e434ca74692 - - - - -] Successfully synced instances from host 'compute01'.</p><p>That's config for nova at compute node:</p><p>[DEFAULT]<br>dhcpbridge_flagfile=/etc/nova/nova.conf<br>dhcpbridge=/usr/bin/nova-dhcpbridge<br>log-dir=/var/log/nova<br>state_path=/var/lib/nova<br>lock_path=/var/lock/nova<br>force_dhcp_release=True<br>libvirt_use_virtio_for_bridges=True<br>debug=true<br>api_paste_config=/etc/nova/api-paste.ini<br>enabled_apis=osapi_compute,metadata<br>rpc_backend = rabbit<br>auth_strategy = keystone<br>my_ip = 10.30.16.129<br>linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver<br>firewall_driver = nova.virt.firewall.NoopFirewallDriver<br>instance_usage_audit = True<br>instance_usage_audit_period = hour<br>notify_on_state_change = vm_and_task_state<br>notification_driver = messagingv2<br>block_device_allocate_retries=28800<br>block_device_allocate_retries_interval=3<br>allow_resize_to_same_host = True<br>[vnc]<br>enabled = True<br>keymap = en-us<br>vncserver_listen = 0.0.0.0<br>vncserver_proxyclient_address = 10.30.16.129<br>novncproxy_base_url = <a href="http://controller:6080/vnc_auto.html" target="_blank">http://controller:6080/vnc_auto.html</a><br>[oslo_messaging_rabbit]<br>rabbit_host = controller<br>rabbit_userid = openstack<br>rabbit_password = PASSWORD<br>[keystone_authtoken]<br>auth_uri = <a href="http://controller:5000/v2.0" target="_blank">http://controller:5000/v2.0</a><br>identity_uri = <a href="http://controller:35357" target="_blank">http://controller:35357</a><br>admin_tenant_name = service<br>admin_user = nova<br>admin_password = PASSWORD</p><p>auth_url = <a href="http://controller:35357" target="_blank">http://controller:35357</a><br>[glance]<br>[oslo_concurrency]<br>lock_path = /var/lib/nova/tmp<br>[neutron]<br>url = <a href="http://controller:9696" target="_blank">http://controller:9696</a><br>auth_strategy = keystone<br>admin_auth_url = <a href="http://controller:35357/v2.0" target="_blank">http://controller:35357/v2.0</a><br>admin_tenant_name = service<br>admin_username = neutron<br>admin_password = PASSWORD<br>[cache]<br>Enabled = true<br>debug_cache_backend = False<br>[libvirt]<br>hw_disk_discard = unmap<br>libvirt_images_rbd_ceph_conf = /etc/ceph/ceph.conf<br>live_migration_flag=VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED,VIR_MIGRATE_NON_SHARED_INC<br>block_migration_flag=VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_LIVE, VIR_MIGRATE_TUNNELLED, VIR_MIGRATE_NON_SHARED_INC<br>disk_cachemodes="network=writeback"</p><p>Neutron's config @compute node:</p><p>[DEFAULT]<br>rpc_backend = rabbit<br>auth_strategy = keystone<br>core_plugin = ml2<br>service_plugins =<br>notify_nova_on_port_status_changes = True<br>notify_nova_on_port_data_changes = True<br>nova_url = <a href="http://controller:8774/v2" target="_blank">http://controller:8774/v2</a><br>verbose = True<br>debug = false<br>[matchmaker_redis]<br>[matchmaker_ring]<br>[quotas]<br>[agent]<br>root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf<br>[keystone_authtoken]<br>auth_uri = <a href="http://controller:5000/v2.0" target="_blank">http://controller:5000/v2.0</a><br>identity_uri = <a href="http://controller:35357" target="_blank">http://controller:35357</a><br>admin_tenant_name = service<br>admin_user = neutron<br>admin_password = PASSWORD</p><p>[database]<br>connection = mysql://neutron:PASSWORD@controller/neutron<br>[nova]<br>auth_url = <a href="http://controller:35357" target="_blank">http://controller:35357</a><br>auth_type = password<br>project_domain_id = default<br>user_domain_id = default<br>region_name = RegionOne<br>project_name = service<br>user-name = nova<br>password = PASSWORD<br>[oslo_concurrency]<br>lock_path = $state_path/lock<br>[oslo_policy]<br>[oslo_messaging_amqp]<br>[oslo_messaging_qpid]<br>[oslo_messaging_rabbit]<br>rabbit_host = controller<br>rabbit_userid = openstack<br>rabbit_password = PASSWORD</p><p>nova's config @controller node:</p><p>[DEFAULT]<br>dhcpbridge_flagfile=/etc/nova/nova.conf<br>dhcpbridge=/usr/bin/nova-dhcpbridge<br>log-dir=/var/log/nova<br>state_path=/var/lib/nova<br>lock_path=/var/lock/nova<br>force_dhcp_release=True<br>libvirt_use_virtio_for_bridges=True<br>debug=false<br>api_paste_config=/etc/nova/api-paste.ini<br>enabled_apis=osapi_compute,metadata<br>rpc_backend = rabbit<br>auth_strategy = keystone<br>my_ip = 10.30.17.10<br>linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver<br>firewall_driver = nova.virt.firewall.NoopFirewallDriver<br>block_device_allocate_retries=28800<br>block_device_allocate_retries_interval=3<br>allow_resize_to_same_host = True<br>[api_database]<br>connection = mysql+pymysql://nova:PASSWORD@controller/nova_api?charset=utf8<br>[database]<br>connection = mysql://nova:PASSWORD@controller/nova<br>[oslo_messaging_rabbit]<br>rabbit_host = controller<br>rabbit_userid = openstack<br>rabbit_password = PASSWORD<br>[keystone_authtoken]<br>auth_uri = <a href="http://controller:5000/v2.0" target="_blank">http://controller:5000/v2.0</a><br>identity_uri = <a href="http://controller:35357" target="_blank">http://controller:35357</a><br>admin_tenant_name = service<br>admin_user = nova<br>admin_password = PASSWORD</p><p>[glance]<br>[oslo_concurrency]<br>lock_path = /var/lib/nova/tmp<br>[neutron]<br>url = <a href="http://controller:9696" target="_blank">http://controller:9696</a><br>auth_strategy = keystone<br>admin_auth_url = <a href="http://controller:35357/v2.0" target="_blank">http://controller:35357/v2.0</a><br>admin_tenant_name = service<br>admin_username = neutron<br>admin_password = PASSWORD<br>service_metadata_proxy = True<br>metadata_proxy_shared_secret = SHARED_SECRET<br>[libvirt]<br>hw_disk_discard = unmap<br>live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED</p><p>Neutron's password @controller node:</p><p>[DEFAULT]<br>rpc_backend = rabbit<br>auth_strategy = keystone<br>core_plugin = ml2<br>service_plugins = router<br>notify_nova_on_port_status_changes = True<br>notify_nova_on_port_data_changes = True<br>nova_url = <a href="http://controller:8774/v2" target="_blank">http://controller:8774/v2</a><br>verbose = True<br>[matchmaker_redis]<br>[matchmaker_ring]<br>[quotas]<br>[agent]<br>root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf<br>[keystone_authtoken]<br>auth_uri = <a href="http://controller:5000/v2.0" target="_blank">http://controller:5000/v2.0</a><br>identity_uri = <a href="http://controller:35357" target="_blank">http://controller:35357</a><br>admin_tenant_name = service<br>admin_user = neutron<br>admin_password = PASSWORD<br>[database]<br>connection = mysql://neutron:PASSWORD@controller/neutron<br>[nova]<br>auth_uri = <a href="http://controller:5000/v2.0" target="_blank">http://controller:5000/v2.0</a><br>identity_uri = <a href="http://controller:35357" target="_blank">http://controller:35357</a><br>admin_tenant_name = service<br>admin_user = nova<br>admin_password = PASSWORD<br>[oslo_concurrency]<br>lock_path = $state_path/lock<br>[oslo_policy]<br>[oslo_messaging_amqp]<br>[oslo_messaging_qpid]<br>[oslo_messaging_rabbit]<br>rabbit_host = controller<br>rabbit_userid = openstack<br>rabbit_password = PASSWORD</p><p>Do you have any idea? Google didn't help (<span class="HOEnZb"><font color="#888888"><br><br><br><br>-- <br>Konstantin</font></span></p></div>
<br>_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to     : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
<br></blockquote></div><br></div>