[Openstack] Devstack - : Failed to allocate the network(s), not rescheduling
Ramu, MohanX
mohanx.ramu at intel.com
Mon Dec 18 15:14:01 UTC 2017
Hi Trinath,
Controller:
ovs-vsctl show
9d006382-2133-4b72-b00b-6333c489a111
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "tap98541bdd-a1"
tag: 1
Interface "tap98541bdd-a1"
type: internal
Port "tap09ec95c9-de"
tag: 3
Interface "tap09ec95c9-de"
type: internal
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Port "qr-0f1e5cc4-72"
tag: 1
Interface "qr-0f1e5cc4-72"
type: internal
Port "qg-786e1436-52"
tag: 2
Interface "qg-786e1436-52"
type: internal
Port "qr-2cf65e0c-73"
tag: 1
Interface "qr-2cf65e0c-73"
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port br-int
Interface br-int
type: internal
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal
Port "vxlan-0a69a62b"
Interface "vxlan-0a69a62b"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="10.105.166.213", out_key=flow, remote_ip="10.105.166.43"}
Bridge br-ex
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
Port br-ex
Interface br-ex
type: internal
ovs_version: "2.6.1"
Compute Node:
root at ubuntu43:~# ovs-vsctl show
b558ac97-64f8-4477-beee-4aed54861205
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port "qvofce55293-8c"
tag: 4095
Interface "qvofce55293-8c"
Port "qvo9e61ed58-88"
tag: 4095
Interface "qvo9e61ed58-88"
Port "qvoa1c640d7-b4"
tag: 5
Interface "qvoa1c640d7-b4"
Port "qvo601239d1-7d"
tag: 4095
Interface "qvo601239d1-7d"
Port "qvocc629c92-4d"
tag: 5
Interface "qvocc629c92-4d"
Port "qvoe4d50bdb-26"
tag: 5
Interface "qvoe4d50bdb-26"
Port "qvod51baf28-aa"
tag: 4095
Interface "qvod51baf28-aa"
Port "qvocde28803-ec"
tag: 5
Interface "qvocde28803-ec"
Port "qvo0be8a7fb-76"
tag: 7
Interface "qvo0be8a7fb-76"
Port "qvo39c05d63-25"
tag: 4095
Interface "qvo39c05d63-25"
Port "qvo7194e36c-99"
tag: 5
Interface "qvoa7eaad53-e7"
Port "qvoa0ca6753-6d"
tag: 4095
Interface "qvoa0ca6753-6d"
Port "qvocf27dd04-26"
tag: 4095
Interface "qvocf27dd04-26"
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal
Port "vxlan-0a69a6d5"
Interface "vxlan-0a69a6d5"
type: vxlan
options: {df_default="true", in_key=flow, local_ip="10.105.166.43", out_key=flow, remote_ip="10.105.166.213"}
Bridge br-ex
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
Port br-ex
Interface br-ex
type: internal
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
ovs_version: "2.6.1"
We are able to launch the instances, but out of 5 one is failing (4:1 ratio). We have modified configuration in the below two files. Both in controller and compute node. When we run ifconfig command it is showing lot of networks and most of them are UP. We never seen these many network list earlier. I am suspecting after adding below configuration under ovs these many networks are coming. Is it a root cause 1 instance is failing out of 5?
Please let us know if you have any idea about it.
1. /etc/neutron/plugins/ml2/ml2_conf.ini
[ml2]
type_drivers = flat
tenant_network_types = flat
mechanism_drivers = openvswitch
[ml2_type_flat]
flat_networks = vmnet
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = True
[ovs]
bridge_mappings = vmnet:br-eno2
2. /opt/stack/neutron/etc/oslo-config-generator/openvswitch_agent.ini
[ovs]
bridge_mappings = vmnet:br-eno2
ovs-vsctl add-br br-int
ovs-vsctl add-br br-eno2
ovs-vsctl add-port br-eno2 eno2
From: Trinath Somanchi [mailto:trinath.somanchi at nxp.com]
Sent: Friday, December 15, 2017 10:59 PM
To: Bernd Bausch <berndbausch at gmail.com>; openstack at lists.openstack.org; Ramu, MohanX <mohanx.ramu at intel.com>
Cc: Sivapuram, Naga SeshendraX <naga.seshendrax.sivapuram at intel.com>
Subject: RE: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling
Can you share the output of below command.
ovs-vsctl show.
________________________________
From: Ramu, MohanX <mohanx.ramu at intel.com<mailto:mohanx.ramu at intel.com>>
Sent: Friday, December 15, 2017 8:41:51 PM
To: Trinath Somanchi; Bernd Bausch; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Cc: Sivapuram, Naga SeshendraX
Subject: RE: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling
Hi Trinath,
We are using flat. Localrc is like local.conf which is shared already.
From: Trinath Somanchi [mailto:trinath.somanchi at nxp.com]
Sent: Friday, December 15, 2017 6:40 PM
To: Ramu, MohanX <mohanx.ramu at intel.com<mailto:mohanx.ramu at intel.com>>; Bernd Bausch <berndbausch at gmail.com<mailto:berndbausch at gmail.com>>; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Cc: Sivapuram, Naga SeshendraX <naga.seshendrax.sivapuram at intel.com<mailto:naga.seshendrax.sivapuram at intel.com>>
Subject: RE: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling
How did you configure the localrc of devstack?
Are you using flat/vlan/vxlan networking with devstack ?
For our flat network based OCATA release openstack setup, where today I fixed this issue - is working fine.
So I shared the fix, so that you can also have neutron-ovs-agent working.
In my OCATA openstack setup, I used neutron-ovs-agent and got the same below mention issue.
/
Trinath Somanchi,
NXP SDN/NFV Team.
From: Ramu, MohanX [mailto:mohanx.ramu at intel.com]
Sent: Friday, December 15, 2017 6:34 PM
To: Trinath Somanchi <trinath.somanchi at nxp.com<mailto:trinath.somanchi at nxp.com>>; Bernd Bausch <berndbausch at gmail.com<mailto:berndbausch at gmail.com>>; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Cc: Sivapuram, Naga SeshendraX <naga.seshendrax.sivapuram at intel.com<mailto:naga.seshendrax.sivapuram at intel.com>>
Subject: RE: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling
Hi Trinath,
We don't have /etc/neutron/plugins/ml2/openvswitch_agent.ini file. We use basic devstack, we don't have any addition into it. Please suggest us what else we should do configuration when we use Devstack set up. We thought that Devstack itself it has all the required configurations. No much idea about Devstack.
But we had another set ups' OpenStack PIKE, OCATA and MITAKA(without devstack), there we have used LinuxBridge agent. It is working perfect.
Thanks & Regards,
Mohan Ramu
From: Trinath Somanchi [mailto:trinath.somanchi at nxp.com]
Sent: Friday, December 15, 2017 5:29 PM
To: Ramu, MohanX <mohanx.ramu at intel.com<mailto:mohanx.ramu at intel.com>>; Bernd Bausch <berndbausch at gmail.com<mailto:berndbausch at gmail.com>>; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Cc: Sivapuram, Naga SeshendraX <naga.seshendrax.sivapuram at intel.com<mailto:naga.seshendrax.sivapuram at intel.com>>
Subject: RE: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling
What is the of_listen_port configured in [ovs] at /etc/neutron/plugins/ml2/openvswitch_agent.ini (or agent config file).
The error, 2017-12-14 21:29:36.326 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
Is caused to bridge configuration issues.
Try, of_listen_port = 7733 and spawn the VMs.
Hope this helps,
/
Trinath Somanchi,
NXP SDN/NFV Team.
From: Ramu, MohanX [mailto:mohanx.ramu at intel.com]
Sent: Friday, December 15, 2017 4:58 PM
To: Bernd Bausch <berndbausch at gmail.com<mailto:berndbausch at gmail.com>>; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Cc: Sivapuram, Naga SeshendraX <naga.seshendrax.sivapuram at intel.com<mailto:naga.seshendrax.sivapuram at intel.com>>
Subject: Re: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling
Hi Bernd,
PFA the local.conf files.
Below are the details used for our devstack environment.
OpenStack version : 15.0.9
Network Used : public
We are not able to launch instances every time. Sometimes instances are launching without any issue. Below are the screenshots for launching multiple instances same time.
Launched 5 instances same time, only 2 instances are created.
[cid:image001.jpg at 01D37840.174BF3D0]
Tried 10 instances same time 5 instances are created.
[cid:image002.jpg at 01D37840.174BF3D0]
Please find logs both in controller and compute side.
q-agt.log (Controller Side) :-
2017-12-14 22:39:32.880 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:37.884 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:37.963 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:42.861 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:47.865 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:47.965 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:48.074 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-12-14 22:39:48.075 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 4 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:48.076 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): DbGetCommand(column=datapath_id, table=Bridge, record=br-int) from (pid=6178) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:100
2017-12-14 22:39:48.077 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction caused no change from (pid=6178) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:124
2017-12-14 22:39:48.078 INFO neutron.plugins.ml2.drive
q-agt.log (Compute Side) :-
2017-12-14 21:29:38.212 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-dadb9cbf-741e-4378-bfc3-9dabf5e78a7d None None] ofctl request version=0x4,msg_type=0x12,msg_len=0x38,xid=0x86ee6f60,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=23,type=1) result [OFPFlowStatsReply(body=[OFPFlowStats(byte_count=0,cookie=9413309945793027316L,duration_nsec=913000000,duration_sec=162395,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],length=56,match=OFPMatch(oxm_fields={}),packet_count=0,priority=0,table_id=23)],flags=0,type=1)] from (pid=517) _send_msg /opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/o2017-12-14 21:29:31.874 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=7063) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:36.090 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=7063) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:36.875 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=7063) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:31.874 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=3621) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:36.090 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=3621) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:36.326 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-12-14 21:29:36.326 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 4 from (pid=3621) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:36.327 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): DbGetCommand(column=datapath_id, table=Bridge, record=br-int) from (pid=3621) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:100
2017-12-14 21:29:36.327 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction caused no change from (pid=3621) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:127
2017-12-14 21:29:36.327 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from de78d015444e to 0000de78d015444e
2017-12-14 21:29:36.875 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=3621) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:31.873 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=815) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:31.875 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=815) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:34.346 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-12-14 21:29:34.346 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 4 from (pid=815) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:34.346 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): DbGetCommand(column=datapath_id, table=Bridge, record=br-int) from (pid=815) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:100
2017-12-14 21:29:34.346 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction caused no change from (pid=815) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:127
2017-12-14 21:29:34.347 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from de78d015444e to 0000de78d015444e
2017-12-14 21:29:36.875 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=815) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
n-cpu.log (Compute Side) :-
2017-12-14 21:24:03.442 DEBUG keystoneauth1.identity.base [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mself.auth_ref ---><keystoneauth1.access.access.AccessInfoV3 object at 0x7ff9d42a2ed0>^[[00m ^[[00;33mfrom (pid=7061) get_access /usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py:138^[[00m^M
2017-12-14 21:24:03.442 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mquery_params --> {}^[[00m^M
2017-12-14 21:24:03.443 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35murl -----::-> http://10.105.166.213/placement/resource_providers/dcf28911-6125-4bc4-87b6-bc71b913d8b2/inventories^[[00m^M<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.105.166.213%2Fplacement%2Fresource_providers%2Fdcf28911-6125-4bc4-87b6-bc71b913d8b2%2Finventories%255e%255b%255b00m%255eM&data=02%7C01%7Ctrinath.somanchi%40nxp.com%7C4cb98e3055544767b83a08d543b021c7%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636489346125753805&sdata=DmzSzlumR5kY0JmoLYS2ghpPSauqJFmVJ7OhXIFfkvU%3D&reserved=0>
2017-12-14 21:24:03.443 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mBefore connection_params^[[00m^M
2017-12-14 21:24:03.443 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mAfter connection_params --> {}^[[00m^M
2017-12-14 21:24:03.443 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mBefore send(**kwargs) ^[[00m^M
2017-12-14 21:24:03.443 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m_send_request call^[[00m^M
2017-12-14 21:24:03.520 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m_send_request ---> <Response [200]>^[[00m^M
2017-12-14 21:24:03.520 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mresp.status_code::: --> (u'http://10.105.166.213/placement/resource_providers/dcf28911-6125-4bc4-87b6-bc71b913d8b2/inventories', 200)^[[00m^M
2017-12-14 21:24:03.520 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mENd of method^[[00m^M
2017-12-14 21:24:03.521 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mresp ---> <Response [200]>^[[00m^M
2017-12-14 21:24:03.521 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mresp.status_code -->(200, u'http://10.105.166.213/placement/resource_providers/dcf28911-6125-4bc4-87b6-bc71b913d8b2/inventories') ^[[00m^M
2017-12-14 21:24:03.521 DEBUG nova.scheduler.client.report [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mUpdating our resource provider generation from 31 to 32^[[00m ^[[00;33mfrom (pid=7061) _get_inventory_and_update_provider_generation /opt/stack/nova/nova/scheduler/client/report.py:449^[[00m^M
2017-12-14 21:24:03.521 DEBUG oslo_concurrency.lockutils [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mLock "compute_resources" released by "nova.compute.resource_tracker.abort_instance_claim" :: held 0.567s^[[00m ^[[00;33mfrom (pid=7061) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282^[[00m^M
2017-12-14 21:24:03.521 ERROR nova.compute.manager [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] Failed to allocate network(s)^[[00m^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00mTraceback (most recent call last):^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m File "/opt/stack/nova/nova/compute/manager.py", line 1938, in _build_and_run_instance^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m with self._build_resources(context, instance,^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2756, in spawn^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m post_xml_callback=gen_confdrive,^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5195, in _create_domain_and_network^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m instance=instance)^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00mVirtualInterfaceCreateException: Virtual Interface creation failed^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m^M
2017-12-14 21:24:03.522 DEBUG nova.compute.utils [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] Virtual Interface creation failed^[[00m ^[[00;33mfrom (pid=7061) notify_about_instance_usage /opt/stack/nova/nova/compute/utils.py:317^[[00m^M
2017-12-14 21:24:03.522 DEBUG oslo_messaging._drivers.amqpdriver [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mCAST unique_id: 9887810c2974467badeac9be672b15d2 NOTIFY exchange 'nova' topic 'notifications.error'^[[00m ^[[00;33mfrom (pid=7061) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:546^[[00m^M
2017-12-14 21:24:03.525 DEBUG oslo_messaging._drivers.amqpdriver [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mCAST unique_id: a59e16bd5ee3492c9db22dd99bd067a3 NOTIFY exchange 'nova' topic 'versioned_notifications.error'^[[00m ^[[00;33mfrom (pid=7061) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:546^[[00m^M
2017-12-14 21:24:03.526 ERROR nova.compute.manager [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] Build of instance 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd aborted: Failed to allocate the network(s), not rescheduling.^[[00m^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00mTraceback (most recent call last):^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m File "/opt/stack/nova/nova/compute/manager.py", line 1788, in _do_build_and_run_instance^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m return build_results.FAILED^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m File "/opt/stack/nova/nova/compute/manager.py", line 1998, in _build_and_run_instance^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m msg = _('Failed to allocate the network(s) with error %s, '^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00mBuildAbortException: Build of instance 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd aborted: Failed to allocate the network(s), not rescheduling.^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m^M
2017-12-14 21:24:03.527 DEBUG nova.compute.manager [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] Deallocating network for instance^[[00m ^[[00;33mfrom (pid=7061) _deallocate_network /opt/stack/nova/nova/compute/manager.py:1669^[[00m^M
2017-12-14 21:24:03.527 DEBUG nova.network.neutronv2.api [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] deallocate_for_instance()^[[00m ^[[00;33mfrom (pid=7061) deallocate_for_instance /opt/stack/nova/nova/network/neutronv2/api.py:1173^[[00m^M
2017-12-14 21:24:03.527 DEBUG keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mauth_headers ----> {'X-Auth-Token': u'gAAAAABaM1TN1hhnQrnfQVASXfFXzyOOq1D8zHS9e1v54TjmkZQiXOSKBcS6kYWTVpDeRKlMHh2GZgOFQ9wJ_gSqvOQ371bTfCk61XV7PR5YeOF9pUP6VQI6x0EojazONSNJJxydImbv1Qf6FISU4HP527itUt_3cpnRnn8-1XE9M6rq_47tIN7fTwwWtkEj6C2x6Djki3Ng'}^[[00m ^[[00;33mfrom (pid=7061) request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:492^[[00m^M
2017-12-14 21:24:03.528 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mquery_params --> {}^[[00m^M
2017-12-14 21:24:03.528 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35murl -----::-> http://10.105.166.213:9696/v2.0/ports.json?device_id=2398fce6-6f98-406d-9b1a-8cf9a1fbfccd^[[00m^M<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.105.166.213%3A9696%2Fv2.0%2Fports.json%3Fdevice_id%3D2398fce6-6f98-406d-9b1a-8cf9a1fbfccd%255e%255b%255b00m%255eM&data=02%7C01%7Ctrinath.somanchi%40nxp.com%7C4cb98e3055544767b83a08d543b021c7%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636489346125753805&sdata=5UJFiEfEtS3pHl3qotjriw2rLPhqTVaQxsswIasW5Jw%3D&reserved=0>
2017-12-14 21:24:03.528 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mBefore connection_params^[[00m^M
2017-12-14 21:24:03.528 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mAfter connection_params --> {}^[[00m^M
2017-12-14 21:24:03.528 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mBefore send(**kwargs) ^[[00m^M
From: Bernd Bausch [mailto:berndbausch at gmail.com]
Sent: Friday, December 15, 2017 6:21 AM
To: Ramu, MohanX <mohanx.ramu at intel.com<mailto:mohanx.ramu at intel.com>>; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Cc: 'Jay Pipes' <jaypipes at gmail.com<mailto:jaypipes at gmail.com>>
Subject: RE: Devstack - : Failed to allocate the network(s), not rescheduling
This just says that Nova was unable to set up the network plumbing on the compute node.
Depending on the OpenStack version you deployed, you should have other messages in the same log file, for example if you are using Pike, "No more network or fixed IP to be allocated" or "Failed to allocate network(s)" (without the article). It would be helpful to know if there are such messages.
Most probably, there is more information in the Neutron logs. Since you use DevStack, you probably have an all-in-one deployment, but in case you have separate compute nodes, also check the Neutron agent log there.
You may also want to provide more information such as: What OpenStack version you have, which log are you sharing, when does this happen, does it always happen or only under certain circumstances, have you ever been able to launch an instance, what networks do you have, what network do you connect the instance to, what's your local.conf etc.
From: Ramu, MohanX [mailto:mohanx.ramu at intel.com]
Sent: Thursday, December 14, 2017 10:54 PM
To: 'openstack at lists.openstack.org' <openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>>
Cc: 'Bernd Bausch' <berndbausch at gmail.com<mailto:berndbausch at gmail.com>>; Jay Pipes <jaypipes at gmail.com<mailto:jaypipes at gmail.com>>
Subject: Devstack - : Failed to allocate the network(s), not rescheduling
Hi All,
2017-12-14 02:56:06.230 DEBUG oslo_messaging._drivers.amqpdriver [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35mCAST unique_id: fa9e604328a445f39702e0089de0248a NOTIFY exchange 'nova' topic 'notifications.error'^[[00m ^[[00;33mfrom (pid=7061) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:546^[[00m^M
2017-12-14 02:56:06.236 DEBUG oslo_messaging._drivers.amqpdriver [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35mCAST unique_id: 999c7e8ac35f4b9db809312ddccc8b7f NOTIFY exchange 'nova' topic 'versioned_notifications.error'^[[00m ^[[00;33mfrom (pid=7061) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:546^[[00m^M
2017-12-14 02:56:06.238 ERROR nova.compute.manager [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] Build of instance 8ed6476c-8c0c-4baf-b24f-8530feb836e7 aborted: Failed to allocate the network(s), not rescheduling.^[[00m^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00mTraceback (most recent call last):^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m File "/opt/stack/nova/nova/compute/manager.py", line 1788, in _do_build_and_run_instance^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m return build_results.FAILED^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m File "/opt/stack/nova/nova/compute/manager.py", line 1998, in _build_and_run_instance^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m msg = _('Failed to allocate the network(s) with error %s, '^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00mBuildAbortException: Build of instance 8ed6476c-8c0c-4baf-b24f-8530feb836e7 aborted: Failed to allocate the network(s), not rescheduling.^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m^M
2017-12-14 02:56:06.239 DEBUG nova.compute.manager [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] Deallocating network for instance^[[00m ^[[00;33mfrom (pid=7061) _deallocate_network /opt/stack/nova/nova/compute/manager.py:1669^[[00m^M
2017-12-14 02:56:06.239 DEBUG nova.network.neutronv2.api [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] deallocate_for_instance()^[[00m ^[[00;33mfrom (pid=7061) deallocate_for_instance /opt/stack/nova/nova/network/neutronv2/api.py:1173^[[00m^M
2017-12-14 02:56:06.240 DEBUG keystoneauth1.session [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35mauth_headers ----> {'X-Auth-Token': u'gAAAAABaMlbpaJ9OocE4phmMPWC6eSmajqtx1id547QxiGEhxMlSddZRexD8TPPH4Il0n3mAemxAWZVXXJeMDAb47ZIlf4YNogzUitsDgkR3-VRSNbB0MdpBFv3aO_aUv5ZBOe1__Ya_bJtW33-QglQbeBqkUoUaQli8aMldZ_BhLBOsqFCYSIWXnd15z73p1j9hmcbxU3Mb'}^[[00m ^[[00;33mfrom (pid=7061) request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:492^[[00m^M
2017-12-14 02:56:06.240 WARNING keystoneauth1.session [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35mquery_params --> {}^[[00m^M
2017-12-14 02:56:06.240 WARNING keystoneauth1.session [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35murl -----::-> http://10.105.166.213:9696/v2.0/ports.json?device_id=8ed6476c-8c0c-4baf-b24f-8530feb836e7^[[00m^M<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.105.166.213%3A9696%2Fv2.0%2Fports.json%3Fdevice_id%3D8ed6476c-8c0c-4baf-b24f-8530feb836e7%255e%255b%255b00m%255eM&data=02%7C01%7Ctrinath.somanchi%40nxp.com%7C4cb98e3055544767b83a08d543b021c7%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636489346125753805&sdata=Ab4DFk8Tfv2p2vN8p0Cm3gaJtydUCNrt8w08Y20HZZQ%3D&reserved=0>
2017-12-14 02:56:06.241 WARNING keystoneauth1.session [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35mBefore connection_params^[[00m^M
2017-12-14 02:56:06.241 WARNING keystoneauth1.session [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35mAfter connection_params --> {}^[[00m^M
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171218/df5dd9f1/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 30036 bytes
Desc: image001.jpg
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171218/df5dd9f1/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 63974 bytes
Desc: image002.jpg
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171218/df5dd9f1/attachment-0001.jpg>
More information about the Openstack
mailing list
