Open Stack

Wed Apr 20 12:07:21 UTC 2016

Hello KK,

I see that your config files are up dated, for example lock_path =
/var/lib/nova/tmp and auth_password to auth_type.
Are the endpoint versions correct? Just taking a wild guess.

Best regards,
Venu

[image: ThoughtWorks] <http://www.thoughtworks.com/>
---
*“Excellence is never an accident. It is always the result of high
intention, sincere effort, and intelligent execution.... ― Aristotle*

On Wed, Apr 20, 2016 at 2:57 PM, K K <nnex at mail.ru> wrote:

> Hello.
>
> I upgraded release OpenStack from Kilo to Mitaka.
>
> But now I can't start any Instance. At the compute node nova daemons said:
>
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> [req-86e4ab2e-229a-438d-87d2-d9ce9f749dbc 79401c3a746740ef826762ca9eaeb207
> 3029462986ac400ca2c2c6ac4c9af5a7 - - -] Exception during message handling:
> Timed out waiting for a reply to message ID cb754491dfb14d8793ca6ee05ae420c9
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> Traceback (most recent call last):
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line
> 138, in _dispatch_and_reply
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> incoming.message))
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line
> 185, in _dispatch
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return
> self._do_dispatch(endpoint, method, ctxt, args)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line
> 127, in _do_dispatch
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result
> = func(ctxt, **new_args)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 110, in wrapped
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher payload)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> self.force_reraise()
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> six.reraise(self.type_, self.value, self.tb)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return
> f(self, context, *args, **kw)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 359, in
> decorated_function
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> LOG.warning(msg, e, instance=instance)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> self.force_reraise()
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> six.reraise(self.type_, self.value, self.tb)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 328, in
> decorated_function
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return
> function(self, context, *args, **kwargs)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in
> decorated_function
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return
> function(self, context, *args, **kwargs)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in
> decorated_function
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> kwargs['instance'], e, sys.exc_info())
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in
> __exit__
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> self.force_reraise()
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in
> force_reraise
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> six.reraise(self.type_, self.value, self.tb)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in
> decorated_function
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return
> function(self, context, *args, **kwargs)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2598, in
> start_instance
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> self._power_on(context, instance)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2566, in
> _power_on
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> network_info = self.network_api.get_instance_nw_info(context, instance)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 49, in wrapped
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return
> func(self, context, *args, **kwargs)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 376, in
> get_instance_nw_info
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> **kwargs)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/network/base_api.py", line 253, in
> get_instance_nw_info
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result
> = self._get_instance_nw_info(context, instance, **kwargs)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 385, in
> _get_instance_nw_info
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher nw_info
> = self.network_rpcapi.get_instance_nw_info(context, **args)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 209, in
> get_instance_nw_info
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> host=host, project_id=project_id)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158,
> in call
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> retry=self.retry)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in
> _send
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> timeout=timeout, retry=retry)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 470, in send
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> retry=retry)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 459, in _send
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result
> = self._waiter.wait(msg_id, timeout)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 342, in wait
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher message
> = self.waiters.get(msg_id, timeout=timeout)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File
> "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
> line 244, in get
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher 'to
> message ID %s' % msg_id)
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> MessagingTimeout: Timed out waiting for a reply to message ID
> cb754491dfb14d8793ca6ee05ae420c9
> 2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher
> 2016-04-20 12:41:15.432 226509 DEBUG oslo_concurrency.lockutils
> [req-793ec582-610d-4702-a975-c642bd7fe799 - - - - -] Releasing semaphore
> "refresh_cache-866156cf-f86b-43c7-9253-68dc1f939c50" lock
> /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228
>
> All services has started up (even rabbitmq at controller node).
> Communications are success between nodes.
>
> At the controller node I have messages:
>
> ==> rabbitmq/rabbit at controller.log <==
>
> =INFO REPORT==== 20-Apr-2016::12:40:16 ===
> accepting AMQP connection <0.3201.0> (10.30.16.129:51314 ->
> 10.30.17.10:5672)
>
> ==> nova/nova-scheduler.log <==
> 2016-04-20 12:40:16.499 4099 INFO nova.scheduler.host_manager
> [req-a010da22-3ea1-4019-82ce-5e434ca74692 - - - - -] Successfully synced
> instances from host 'compute01'.
>
> That's config for nova at compute node:
>
> [DEFAULT]
> dhcpbridge_flagfile=/etc/nova/nova.conf
> dhcpbridge=/usr/bin/nova-dhcpbridge
> log-dir=/var/log/nova
> state_path=/var/lib/nova
> lock_path=/var/lock/nova
> force_dhcp_release=True
> libvirt_use_virtio_for_bridges=True
> debug=true
> api_paste_config=/etc/nova/api-paste.ini
> enabled_apis=osapi_compute,metadata
> rpc_backend = rabbit
> auth_strategy = keystone
> my_ip = 10.30.16.129
> linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
> firewall_driver = nova.virt.firewall.NoopFirewallDriver
> instance_usage_audit = True
> instance_usage_audit_period = hour
> notify_on_state_change = vm_and_task_state
> notification_driver = messagingv2
> block_device_allocate_retries=28800
> block_device_allocate_retries_interval=3
> allow_resize_to_same_host = True
> [vnc]
> enabled = True
> keymap = en-us
> vncserver_listen = 0.0.0.0
> vncserver_proxyclient_address = 10.30.16.129
> novncproxy_base_url = http://controller:6080/vnc_auto.html
> [oslo_messaging_rabbit]
> rabbit_host = controller
> rabbit_userid = openstack
> rabbit_password = PASSWORD
> [keystone_authtoken]
> auth_uri = http://controller:5000/v2.0
> identity_uri = http://controller:35357
> admin_tenant_name = service
> admin_user = nova
> admin_password = PASSWORD
>
> auth_url = http://controller:35357
> [glance]
> [oslo_concurrency]
> lock_path = /var/lib/nova/tmp
> [neutron]
> url = http://controller:9696
> auth_strategy = keystone
> admin_auth_url = http://controller:35357/v2.0
> admin_tenant_name = service
> admin_username = neutron
> admin_password = PASSWORD
> [cache]
> Enabled = true
> debug_cache_backend = False
> [libvirt]
> hw_disk_discard = unmap
> libvirt_images_rbd_ceph_conf = /etc/ceph/ceph.conf
>
> live_migration_flag=VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED,VIR_MIGRATE_NON_SHARED_INC
> block_migration_flag=VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_LIVE,
> VIR_MIGRATE_TUNNELLED, VIR_MIGRATE_NON_SHARED_INC
> disk_cachemodes="network=writeback"
>
> Neutron's config @compute node:
>
> [DEFAULT]
> rpc_backend = rabbit
> auth_strategy = keystone
> core_plugin = ml2
> service_plugins =
> notify_nova_on_port_status_changes = True
> notify_nova_on_port_data_changes = True
> nova_url = http://controller:8774/v2
> verbose = True
> debug = false
> [matchmaker_redis]
> [matchmaker_ring]
> [quotas]
> [agent]
> root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
> [keystone_authtoken]
> auth_uri = http://controller:5000/v2.0
> identity_uri = http://controller:35357
> admin_tenant_name = service
> admin_user = neutron
> admin_password = PASSWORD
>
> [database]
> connection = mysql://neutron:PASSWORD@controller/neutron
> [nova]
> auth_url = http://controller:35357
> auth_type = password
> project_domain_id = default
> user_domain_id = default
> region_name = RegionOne
> project_name = service
> user-name = nova
> password = PASSWORD
> [oslo_concurrency]
> lock_path = $state_path/lock
> [oslo_policy]
> [oslo_messaging_amqp]
> [oslo_messaging_qpid]
> [oslo_messaging_rabbit]
> rabbit_host = controller
> rabbit_userid = openstack
> rabbit_password = PASSWORD
>
> nova's config @controller node:
>
> [DEFAULT]
> dhcpbridge_flagfile=/etc/nova/nova.conf
> dhcpbridge=/usr/bin/nova-dhcpbridge
> log-dir=/var/log/nova
> state_path=/var/lib/nova
> lock_path=/var/lock/nova
> force_dhcp_release=True
> libvirt_use_virtio_for_bridges=True
> debug=false
> api_paste_config=/etc/nova/api-paste.ini
> enabled_apis=osapi_compute,metadata
> rpc_backend = rabbit
> auth_strategy = keystone
> my_ip = 10.30.17.10
> linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
> firewall_driver = nova.virt.firewall.NoopFirewallDriver
> block_device_allocate_retries=28800
> block_device_allocate_retries_interval=3
> allow_resize_to_same_host = True
> [api_database]
> connection = mysql+pymysql://nova:PASSWORD@controller
> /nova_api?charset=utf8
> [database]
> connection = mysql://nova:PASSWORD@controller/nova
> [oslo_messaging_rabbit]
> rabbit_host = controller
> rabbit_userid = openstack
> rabbit_password = PASSWORD
> [keystone_authtoken]
> auth_uri = http://controller:5000/v2.0
> identity_uri = http://controller:35357
> admin_tenant_name = service
> admin_user = nova
> admin_password = PASSWORD
>
> [glance]
> [oslo_concurrency]
> lock_path = /var/lib/nova/tmp
> [neutron]
> url = http://controller:9696
> auth_strategy = keystone
> admin_auth_url = http://controller:35357/v2.0
> admin_tenant_name = service
> admin_username = neutron
> admin_password = PASSWORD
> service_metadata_proxy = True
> metadata_proxy_shared_secret = SHARED_SECRET
> [libvirt]
> hw_disk_discard = unmap
>
> live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED
>
> Neutron's password @controller node:
>
> [DEFAULT]
> rpc_backend = rabbit
> auth_strategy = keystone
> core_plugin = ml2
> service_plugins = router
> notify_nova_on_port_status_changes = True
> notify_nova_on_port_data_changes = True
> nova_url = http://controller:8774/v2
> verbose = True
> [matchmaker_redis]
> [matchmaker_ring]
> [quotas]
> [agent]
> root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
> [keystone_authtoken]
> auth_uri = http://controller:5000/v2.0
> identity_uri = http://controller:35357
> admin_tenant_name = service
> admin_user = neutron
> admin_password = PASSWORD
> [database]
> connection = mysql://neutron:PASSWORD@controller/neutron
> [nova]
> auth_uri = http://controller:5000/v2.0
> identity_uri = http://controller:35357
> admin_tenant_name = service
> admin_user = nova
> admin_password = PASSWORD
> [oslo_concurrency]
> lock_path = $state_path/lock
> [oslo_policy]
> [oslo_messaging_amqp]
> [oslo_messaging_qpid]
> [oslo_messaging_rabbit]
> rabbit_host = controller
> rabbit_userid = openstack
> rabbit_password = PASSWORD
>
> Do you have any idea? Google didn't help (
>
>
>
> --
> Konstantin
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160420/b8921fac/attachment.html>

Open Stack

[Openstack] Can't start instance after upgrade to Mitaka release

OpenStack

Community

Documentation

Branding & Legal