[Openstack] launching instance failed since of nova-network quantum
Vaze, Mandar
Mandar.Vaze at nttdata.com
Wed May 2 12:56:56 UTC 2012
I too have seen occasional rpc timeout errors recently, (never till essex) I also use Quantum.
But I thought this may have to do with my local setup, changes etc. I’ll pay more attention if I see it again
-Mandar
From: openstack-bounces+mandar.vaze=nttdata.com at lists.launchpad.net [mailto:openstack-bounces+mandar.vaze=nttdata.com at lists.launchpad.net] On Behalf Of livemoon
Sent: Wednesday, May 02, 2012 11:13 AM
To: openstack at lists.launchpad.net
Subject: [Openstack] launching instance failed since of nova-network quantum
Hi, all
I use quantum and OVS in ubuntu 12.04 ,when I lanuch an instance, it return networking error.
This is nova-network log:
2012-05-02 13:26:38 DEBUG nova.utils [req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45 5567b78b1a3f4bc68421fe74e044f4e3] Attempting to grab semaphore "quantum-enable-dhcp" for method "enable_dhcp"... from (pid=31609) inner /usr/lib/python2.7/dist-packages/nova/utils.py:927
2012-05-02 13:26:38 DEBUG nova.utils [req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45 5567b78b1a3f4bc68421fe74e044f4e3] Got semaphore "quantum-enable-dhcp" for method "enable_dhcp"... from (pid=31609) inner /usr/lib/python2.7/dist-packages/nova/utils.py:931
2012-05-02 13:26:38 INFO nova.network.quantum.manager [req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45 5567b78b1a3f4bc68421fe74e044f4e3] Using DHCP for network: test-1
2012-05-02 13:26:38 DEBUG nova.network.quantum.quantum_connection [req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45 5567b78b1a3f4bc68421fe74e044f4e3] Quantum Client Request: GET /v1.1/tenants/5567b78b1a3f4bc68421fe74e044f4e3/networks/b7eb78f2-b4c2-4106-99c0-0b99f58b2e1e/ports.json?attachment=gw-b7eb78f2-b4 from (pid=31609) do_request /usr/lib/python2.7/dist-packages/nova/network/quantum/client.py:181
2012-05-02 13:26:38 DEBUG nova.network.quantum.quantum_connection [req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45 5567b78b1a3f4bc68421fe74e044f4e3] Quantum Client Reply (code = 200) :
{"ports": []} from (pid=31609) do_request /usr/lib/python2.7/dist-packages/nova/network/quantum/client.py:192
2012-05-02 13:26:38 DEBUG nova.utils [req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45 5567b78b1a3f4bc68421fe74e044f4e3] Running cmd (subprocess): ip link show dev gw-b7eb78f2-b4 from (pid=31609) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219
2012-05-02 13:26:38 DEBUG nova.utils [req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45 5567b78b1a3f4bc68421fe74e044f4e3] Result was 1 from (pid=31609) execute /usr/lib/python2.7/dist-packages/nova/utils.py:235
2012-05-02 13:26:38 DEBUG nova.utils [req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45 5567b78b1a3f4bc68421fe74e044f4e3] Running cmd (subprocess): sudo nova-rootwrap ovs-vsctl -- --may-exist add-port br-int gw-b7eb78f2-b4 -- set Interface gw-b7eb78f2-b4 type=internal -- set Interface gw-b7eb78f2-b4 external-ids:iface-id=gw-b7eb78f2-b4 -- set Interface gw-b7eb78f2-b4 external-ids:iface-status=active -- set Interface gw-b7eb78f2-b4 external-ids:attached-mac=fa:16:3e:6f:b7:38 from (pid=31609) execute /usr/lib/python2.7/dist-packages/nova/utils.py:219
2012-05-02 13:27:16 DEBUG nova.manager [req-91974e5f-cea9-44a3-a4e3-ef5955aee599 None None] Running periodic task QuantumManager._publish_service_capabilities from (pid=31609) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152
2012-05-02 13:27:16 DEBUG nova.manager [req-91974e5f-cea9-44a3-a4e3-ef5955aee599 None None] Running periodic task QuantumManager._disassociate_stale_fixed_ips from (pid=31609) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152
2012-05-02 13:27:23 DEBUG nova.rpc.amqp [-] received {u'_context_roles': [u'admin'], u'_msg_id': u'7584dd267c8f47caa02927c88912383a', u'_context_read_deleted': u'no', u'_context_request_id': u'req-9e8a3e43-f55c-4a1e-8565-892708a85338', u'args': {u'instance_id': 2, u'instance_uuid': u'dc18235c-977f-4fd2-8828-b98155ce2307', u'host': u'cloud', u'project_id': u'5567b78b1a3f4bc68421fe74e044f4e3', u'rxtx_factor': 1.0}, u'_context_auth_token': '<SANITIZED>', u'_context_is_admin': True, u'_context_project_id': None, u'_context_timestamp': u'2012-05-02T05:27:22.702310', u'_context_user_id': None, u'method': u'get_instance_nw_info', u'_context_remote_address': None} from (pid=31609) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160
2012-05-02 13:27:23 DEBUG nova.rpc.amqp [req-9e8a3e43-f55c-4a1e-8565-892708a85338 None None] unpacked context: {'user_id': None, 'roles': [u'admin'], 'timestamp': '2012-05-02T05:27:22.702310', 'auth_token': '<SANITIZED>', 'remote_address': None, 'is_admin': True, 'request_id': u'req-9e8a3e43-f55c-4a1e-8565-892708a85338', 'project_id': None, 'read_deleted': u'no'} from (pid=31609) _safe_log /usr/lib/python2.7/dist-packages/nova/rpc/common.py:160
This is nova-compute log:
2012-05-02 13:27:23 DEBUG nova.virt.libvirt.connection [req-9e8a3e43-f55c-4a1e-8565-892708a85338 None None] Connecting to libvirt: qemu:///system from (pid=31711) _get_connection /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py:292
2012-05-02 13:27:23 DEBUG nova.virt.libvirt.connection [req-9e8a3e43-f55c-4a1e-8565-892708a85338 None None] Updating host stats from (pid=31711) update_status /usr/lib/python2.7/dist-packages/nova/virt/libvirt/connection.py:2467
2012-05-02 13:27:23 DEBUG nova.manager [req-9e8a3e43-f55c-4a1e-8565-892708a85338 None None] Running periodic task ComputeManager._poll_unconfirmed_resizes from (pid=31711) periodic_tasks /usr/lib/python2.7/dist-packages/nova/manager.py:152
2012-05-02 13:27:38 ERROR nova.rpc.common [req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45 5567b78b1a3f4bc68421fe74e044f4e3] Timed out waiting for RPC response: timed out
2012-05-02 13:27:38 TRACE nova.rpc.common Traceback (most recent call last):
2012-05-02 13:27:38 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 490, in ensure
2012-05-02 13:27:38 TRACE nova.rpc.common return method(*args, **kwargs)
2012-05-02 13:27:38 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 567, in _consume
2012-05-02 13:27:38 TRACE nova.rpc.common return self.connection.drain_events(timeout=timeout)
2012-05-02 13:27:38 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 175, in drain_events
2012-05-02 13:27:38 TRACE nova.rpc.common return self.transport.drain_events(self.connection, **kwargs)
2012-05-02 13:27:38 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 238, in drain_events
2012-05-02 13:27:38 TRACE nova.rpc.common return connection.drain_events(**kwargs)
2012-05-02 13:27:38 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 57, in drain_events
2012-05-02 13:27:38 TRACE nova.rpc.common return self.wait_multi(self.channels.values(), timeout=timeout)
2012-05-02 13:27:38 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 63, in wait_multi
2012-05-02 13:27:38 TRACE nova.rpc.common chanmap.keys(), allowed_methods, timeout=timeout)
2012-05-02 13:27:38 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 120, in _wait_multiple
2012-05-02 13:27:38 TRACE nova.rpc.common channel, method_sig, args, content = read_timeout(timeout)
2012-05-02 13:27:38 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqplib.py", line 94, in read_timeout
2012-05-02 13:27:38 TRACE nova.rpc.common return self.method_reader.read_method()
2012-05-02 13:27:38 TRACE nova.rpc.common File "/usr/lib/python2.7/dist-packages/amqplib/client_0_8/method_framing.py", line 221, in read_method
2012-05-02 13:27:38 TRACE nova.rpc.common raise m
2012-05-02 13:27:38 TRACE nova.rpc.common timeout: timed out
2012-05-02 13:27:38 TRACE nova.rpc.common
2012-05-02 13:27:38 ERROR nova.compute.manager [req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45 5567b78b1a3f4bc68421fe74e044f4e3] [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] Instance failed network setup
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] Traceback (most recent call last):
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 559, in _allocate_network
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] requested_networks=requested_networks)
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 170, in allocate_for_instance
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] 'args': args})
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] File "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 68, in call
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] return _get_impl().call(context, topic, msg, timeout)
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 674, in call
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 338, in call
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] rv = list(rv)
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 299, in __iter__
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] self._iterator.next()
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 572, in iterconsume
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] yield self.ensure(_error_callback, _consume)
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 503, in ensure
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] error_callback(e)
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 553, in _error_callback
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] raise rpc_common.Timeout()
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01] Timeout: Timeout while waiting on RPC response.
2012-05-02 13:27:38 TRACE nova.compute.manager [instance: 8fc1536d-3681-4d13-9211-dc88c4dc2b01]
2012-05-02 13:27:38 ERROR nova.rpc.amqp [req-3c1ab45e-7ea3-4e83-a899-ab1a8dbde09d a372a6b75d254b15b83cc203518a8a45 5567b78b1a3f4bc68421fe74e044f4e3] Exception during message handling
2012-05-02 13:27:38 TRACE nova.rpc.amqp Traceback (most recent call last):
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 252, in _process_data
2012-05-02 13:27:38 TRACE nova.rpc.amqp rval = node_func(context=ctxt, **node_args)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/exception.py", line 114, in wrapped
2012-05-02 13:27:38 TRACE nova.rpc.amqp return f(*args, **kw)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 177, in decorated_function
2012-05-02 13:27:38 TRACE nova.rpc.amqp sys.exc_info())
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-05-02 13:27:38 TRACE nova.rpc.amqp self.gen.next()
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 171, in decorated_function
2012-05-02 13:27:38 TRACE nova.rpc.amqp return function(self, context, instance_uuid, *args, **kwargs)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 651, in run_instance
2012-05-02 13:27:38 TRACE nova.rpc.amqp do_run_instance()
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/utils.py", line 945, in inner
2012-05-02 13:27:38 TRACE nova.rpc.amqp retval = f(*args, **kwargs)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 650, in do_run_instance
2012-05-02 13:27:38 TRACE nova.rpc.amqp self._run_instance(context, instance_uuid, **kwargs)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 451, in _run_instance
2012-05-02 13:27:38 TRACE nova.rpc.amqp self._set_instance_error_state(context, instance_uuid)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/contextlib.py", line 24, in __exit__
2012-05-02 13:27:38 TRACE nova.rpc.amqp self.gen.next()
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 424, in _run_instance
2012-05-02 13:27:38 TRACE nova.rpc.amqp requested_networks)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/compute/manager.py", line 559, in _allocate_network
2012-05-02 13:27:38 TRACE nova.rpc.amqp requested_networks=requested_networks)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 170, in allocate_for_instance
2012-05-02 13:27:38 TRACE nova.rpc.amqp 'args': args})
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/__init__.py", line 68, in call
2012-05-02 13:27:38 TRACE nova.rpc.amqp return _get_impl().call(context, topic, msg, timeout)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 674, in call
2012-05-02 13:27:38 TRACE nova.rpc.amqp return rpc_amqp.call(context, topic, msg, timeout, Connection.pool)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 338, in call
2012-05-02 13:27:38 TRACE nova.rpc.amqp rv = list(rv)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/amqp.py", line 299, in __iter__
2012-05-02 13:27:38 TRACE nova.rpc.amqp self._iterator.next()
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 572, in iterconsume
2012-05-02 13:27:38 TRACE nova.rpc.amqp yield self.ensure(_error_callback, _consume)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 503, in ensure
2012-05-02 13:27:38 TRACE nova.rpc.amqp error_callback(e)
2012-05-02 13:27:38 TRACE nova.rpc.amqp File "/usr/lib/python2.7/dist-packages/nova/rpc/impl_kombu.py", line 553, in _error_callback
2012-05-02 13:27:38 TRACE nova.rpc.amqp raise rpc_common.Timeout()
2012-05-02 13:27:38 TRACE nova.rpc.amqp Timeout: Timeout while waiting on RPC response.
2012-05-02 13:27:38 TRACE nova.rpc.amqp
my nova.conf is
--dhcpbridge_flagfile=/etc/nova/nova.conf
--dhcpbridge=/usr/bin/nova-dhcpbridge
--logdir=/var/log/nova
--state_path=/var/lib/nova
--lock_path=/var/lock/nova
--force_dhcp_release
--iscsi_helper=tgtadm
--libvirt_use_virtio_for_bridges
--connection_type=libvirt
--root_helper=sudo nova-rootwrap
--verbose
--ec2_private_dns_show_ip
--network_manager=nova.network.quantum.manager.QuantumManager
--linuxnet_interface_driver=nova.network.linux_net.LinuxOVSInterfaceDriver
--libvirt_ovs_bridge=br-int
--libvirt_vif_type=ethernet
--libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
--quantum_use_dhcp=True
--quantum_connection_host=localhost
--quantum_connection_port=9696
--scheduler_driver=nova.scheduler.simple.SimpleScheduler
#--fixed_range=10.0.0.0/16<http://10.0.0.0/16>
--s3_host=172.16.0.159
--osapi_compute_extension=nova.api.openstack.compute.contrib.standard_extensions
--my_ip=172.16.0.159
--public_interface=eth0
#--vlan_interface=
#--flat_network_bridge=br100
#--flat_interface=eth1
--sql_connection=mysql://novadbadmin:123456@localhost/nova
--instance_name_template=instance-%08x
--novncproxy_base_url=http://172.16.0.159:6080/vnc_auto.html
--xvpvncproxy_base_url=http://172.16.0.159:6081/console
--vncserver_listen=172.16.0.159
--vncserver_proxyclient_address=172.16.0.159
--api_paste_config=/etc/nova/api-paste.ini
--image_service=nova.image.glance.GlanceImageService
--ec2_dmz_host=172.16.0.159
--rabbit_host=172.16.0.159
--rabbit_password=guest
--glance_api_servers=172.16.0.159:9292<http://172.16.0.159:9292>
--force_dhcp_release
--send_arp_for_ha
--libvirt_use_virtio_for_bridges
--firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
--auth_strategy=keystone
my nova-manage network list is:
id IPv4 IPv6 start address DNS1 DNS2 VlanID project uuid
6 10.0.1.0/24<http://10.0.1.0/24> None 10.0.1.2 8.8.4.4 None None None 56136df2-4644-4d68-bec1-d9285cfa9186
7 10.0.2.0/24<http://10.0.2.0/24> None 10.0.2.2 8.8.4.4 None None 5567b78b1a3f4bc68421fe74e044f4e3 b7eb78f2-b4c2-4106-99c0-0b99f58b2e1e
ovs network is:
root at cloud:~# ovs-vsctl show
3f474726-21fc-40a4-b7b8-f438fcfe2c3e
Bridge br-int
Port "eth1"
Interface "eth1"
Port br-int
Interface br-int
type: internal
--
非淡薄无以明志，非宁静无以致远
______________________________________________________________________
Disclaimer:This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20120502/348c73c9/attachment.html>
More information about the Openstack
mailing list