Open Stack

Hello.
I upgraded release OpenStack from Kilo to Mitaka.
But now I can't start any Instance. At the compute node nova daemons said:
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher [req-86e4ab2e-229a-438d-87d2-d9ce9f749dbc 79401c3a746740ef826762ca9eaeb207 3029462986ac400ca2c2c6ac4c9af5a7 - - -] Exception during message handling: Timed out waiting for a reply to message ID cb754491dfb14d8793ca6ee05ae420c9
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 138, in _dispatch_and_reply
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher incoming.message))
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 185, in _dispatch
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return self._do_dispatch(endpoint, method, ctxt, args)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 127, in _do_dispatch
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result = func(ctxt, **new_args)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 110, in wrapped
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher payload)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 89, in wrapped
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return f(self, context, *args, **kw)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 359, in decorated_function
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher LOG.warning(msg, e, instance=instance)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 328, in decorated_function
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 409, in decorated_function
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 387, in decorated_function
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher kwargs['instance'], e, sys.exc_info())
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 220, in __exit__
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher self.force_reraise()
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher six.reraise(self.type_, self.value, self.tb)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 375, in decorated_function
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return function(self, context, *args, **kwargs)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2598, in start_instance
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher self._power_on(context, instance)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 2566, in _power_on
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher network_info = self.network_api.get_instance_nw_info(context, instance)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 49, in wrapped
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher return func(self, context, *args, **kwargs)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 376, in get_instance_nw_info
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher **kwargs)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/base_api.py", line 253, in get_instance_nw_info
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result = self._get_instance_nw_info(context, instance, **kwargs)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 385, in _get_instance_nw_info
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher nw_info = self.network_rpcapi.get_instance_nw_info(context, **args)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 209, in get_instance_nw_info
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher host=host, project_id=project_id)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 158, in call
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher retry=self.retry)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 90, in _send
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher timeout=timeout, retry=retry)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher retry=retry)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 459, in _send
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher result = self._waiter.wait(msg_id, timeout)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 342, in wait
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher message = self.waiters.get(msg_id, timeout=timeout)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 244, in get
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher 'to message ID %s' % msg_id)
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher MessagingTimeout: Timed out waiting for a reply to message ID cb754491dfb14d8793ca6ee05ae420c9
2016-04-20 12:40:15.395 226509 ERROR oslo_messaging.rpc.dispatcher 
2016-04-20 12:41:15.432 226509 DEBUG oslo_concurrency.lockutils [req-793ec582-610d-4702-a975-c642bd7fe799 - - - - -] Releasing semaphore "refresh_cache-866156cf-f86b-43c7-9253-68dc1f939c50" lock /usr/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:228
All services has started up (even rabbitmq at controller node). Communications are success between nodes.
At the controller node I have messages:
==> rabbitmq/rabbit at controller.log <==
=INFO REPORT==== 20-Apr-2016::12:40:16 ===
accepting AMQP connection <0.3201.0> (10.30.16.129:51314 -> 10.30.17.10:5672)
==> nova/nova-scheduler.log <==
2016-04-20 12:40:16.499 4099 INFO nova.scheduler.host_manager [req-a010da22-3ea1-4019-82ce-5e434ca74692 - - - - -] Successfully synced instances from host 'compute01'.
That's config for nova at compute node:
[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
log-dir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
libvirt_use_virtio_for_bridges=True
debug=true
api_paste_config=/etc/nova/api-paste.ini
enabled_apis=osapi_compute,metadata
rpc_backend = rabbit
auth_strategy = keystone
my_ip = 10.30.16.129
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
firewall_driver = nova.virt.firewall.NoopFirewallDriver
instance_usage_audit = True
instance_usage_audit_period = hour
notify_on_state_change = vm_and_task_state
notification_driver = messagingv2
block_device_allocate_retries=28800
block_device_allocate_retries_interval=3
allow_resize_to_same_host = True
[vnc]
enabled = True
keymap = en-us
vncserver_listen = 0.0.0.0
vncserver_proxyclient_address = 10.30.16.129
novncproxy_base_url = http://controller:6080/vnc_auto.html
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = PASSWORD
[keystone_authtoken]
auth_uri = http://controller:5000/v2.0
identity_uri = http://controller:35357
admin_tenant_name = service
admin_user = nova
admin_password = PASSWORD
auth_url = http://controller:35357
[glance]
[oslo_concurrency]
lock_path = /var/lib/nova/tmp
[neutron]
url = http://controller:9696
auth_strategy = keystone
admin_auth_url = http://controller:35357/v2.0
admin_tenant_name = service
admin_username = neutron
admin_password = PASSWORD
[cache]
Enabled = true
debug_cache_backend = False
[libvirt]
hw_disk_discard = unmap
libvirt_images_rbd_ceph_conf = /etc/ceph/ceph.conf
live_migration_flag=VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED,VIR_MIGRATE_NON_SHARED_INC
block_migration_flag=VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_LIVE, VIR_MIGRATE_TUNNELLED, VIR_MIGRATE_NON_SHARED_INC
disk_cachemodes="network=writeback"
Neutron's config @compute node:
[DEFAULT]
rpc_backend = rabbit
auth_strategy = keystone
core_plugin = ml2
service_plugins =
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
nova_url = http://controller:8774/v2
verbose = True
debug = false
[matchmaker_redis]
[matchmaker_ring]
[quotas]
[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
[keystone_authtoken]
auth_uri = http://controller:5000/v2.0
identity_uri = http://controller:35357
admin_tenant_name = service
admin_user = neutron
admin_password = PASSWORD
[database]
connection = mysql://neutron:PASSWORD@controller/neutron
[nova]
auth_url = http://controller:35357
auth_type = password
project_domain_id = default
user_domain_id = default
region_name = RegionOne
project_name = service
user-name = nova
password = PASSWORD
[oslo_concurrency]
lock_path = $state_path/lock
[oslo_policy]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = PASSWORD
nova's config @controller node:
[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
log-dir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
libvirt_use_virtio_for_bridges=True
debug=false
api_paste_config=/etc/nova/api-paste.ini
enabled_apis=osapi_compute,metadata
rpc_backend = rabbit
auth_strategy = keystone
my_ip = 10.30.17.10
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
firewall_driver = nova.virt.firewall.NoopFirewallDriver
block_device_allocate_retries=28800
block_device_allocate_retries_interval=3
allow_resize_to_same_host = True
[api_database]
connection = mysql+pymysql://nova:PASSWORD@controller/nova_api?charset=utf8
[database]
connection = mysql://nova:PASSWORD@controller/nova
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = PASSWORD
[keystone_authtoken]
auth_uri = http://controller:5000/v2.0
identity_uri = http://controller:35357
admin_tenant_name = service
admin_user = nova
admin_password = PASSWORD
[glance]
[oslo_concurrency]
lock_path = /var/lib/nova/tmp
[neutron]
url = http://controller:9696
auth_strategy = keystone
admin_auth_url = http://controller:35357/v2.0
admin_tenant_name = service
admin_username = neutron
admin_password = PASSWORD
service_metadata_proxy = True
metadata_proxy_shared_secret = SHARED_SECRET
[libvirt]
hw_disk_discard = unmap
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE,VIR_MIGRATE_TUNNELLED
Neutron's password @controller node:
[DEFAULT]
rpc_backend = rabbit
auth_strategy = keystone
core_plugin = ml2
service_plugins = router
notify_nova_on_port_status_changes = True
notify_nova_on_port_data_changes = True
nova_url = http://controller:8774/v2
verbose = True
[matchmaker_redis]
[matchmaker_ring]
[quotas]
[agent]
root_helper = sudo /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf
[keystone_authtoken]
auth_uri = http://controller:5000/v2.0
identity_uri = http://controller:35357
admin_tenant_name = service
admin_user = neutron
admin_password = PASSWORD
[database]
connection = mysql://neutron:PASSWORD@controller/neutron
[nova]
auth_uri = http://controller:5000/v2.0
identity_uri = http://controller:35357
admin_tenant_name = service
admin_user = nova
admin_password = PASSWORD
[oslo_concurrency]
lock_path = $state_path/lock
[oslo_policy]
[oslo_messaging_amqp]
[oslo_messaging_qpid]
[oslo_messaging_rabbit]
rabbit_host = controller
rabbit_userid = openstack
rabbit_password = PASSWORD
Do you have any idea? Google didn't help (



-- 
Konstantin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160420/88a00de2/attachment.html>