[Openstack] Devstack - : Failed to allocate the network(s), not rescheduling

Trinath Somanchi trinath.somanchi at nxp.com
Tue Dec 19 06:01:00 UTC 2017


"We are able to launch the instances, but out of 5 one is failing (4:1 ratio)." - Did you verify the nova and neutron-agent logs to trace the reason of failure?

/
   Trinath Somanchi | HSDC | NXP INDIA

From: Ramu, MohanX [mailto:mohanx.ramu at intel.com]
Sent: Monday, December 18, 2017 8:44 PM
To: Trinath Somanchi <trinath.somanchi at nxp.com>; Bernd Bausch <berndbausch at gmail.com>; openstack at lists.openstack.org
Cc: Sivapuram, Naga SeshendraX <naga.seshendrax.sivapuram at intel.com>
Subject: RE: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling

Hi Trinath,

Controller:

ovs-vsctl show
9d006382-2133-4b72-b00b-6333c489a111
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
       Port "tap98541bdd-a1"
            tag: 1
            Interface "tap98541bdd-a1"
                type: internal
        Port "tap09ec95c9-de"
            tag: 3
            Interface "tap09ec95c9-de"
                type: internal
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
        Port "qr-0f1e5cc4-72"
            tag: 1
            Interface "qr-0f1e5cc4-72"
                type: internal
        Port "qg-786e1436-52"
           tag: 2
            Interface "qg-786e1436-52"
                type: internal
        Port "qr-2cf65e0c-73"
            tag: 1
            Interface "qr-2cf65e0c-73"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
        Port br-int
            Interface br-int
                type: internal
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-0a69a62b"
            Interface "vxlan-0a69a62b"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.105.166.213", out_key=flow, remote_ip="10.105.166.43"}
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
        Port br-ex
            Interface br-ex
                type: internal
    ovs_version: "2.6.1"

Compute Node:

root at ubuntu43:~# ovs-vsctl show
b558ac97-64f8-4477-beee-4aed54861205
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port "qvofce55293-8c"
            tag: 4095
            Interface "qvofce55293-8c"
        Port "qvo9e61ed58-88"
            tag: 4095
            Interface "qvo9e61ed58-88"
        Port "qvoa1c640d7-b4"
            tag: 5
            Interface "qvoa1c640d7-b4"
        Port "qvo601239d1-7d"
            tag: 4095
            Interface "qvo601239d1-7d"
        Port "qvocc629c92-4d"
            tag: 5
            Interface "qvocc629c92-4d"
        Port "qvoe4d50bdb-26"
            tag: 5
            Interface "qvoe4d50bdb-26"
        Port "qvod51baf28-aa"
            tag: 4095
            Interface "qvod51baf28-aa"
        Port "qvocde28803-ec"
            tag: 5
            Interface "qvocde28803-ec"
        Port "qvo0be8a7fb-76"
            tag: 7
            Interface "qvo0be8a7fb-76"
        Port "qvo39c05d63-25"
            tag: 4095
            Interface "qvo39c05d63-25"
        Port "qvo7194e36c-99"
            tag: 5

Interface "qvoa7eaad53-e7"
        Port "qvoa0ca6753-6d"
            tag: 4095
            Interface "qvoa0ca6753-6d"
        Port "qvocf27dd04-26"
            tag: 4095
            Interface "qvocf27dd04-26"
        Port int-br-ex
            Interface int-br-ex
                type: patch
                options: {peer=phy-br-ex}
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port br-tun
            Interface br-tun
                type: internal
        Port "vxlan-0a69a6d5"
            Interface "vxlan-0a69a6d5"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.105.166.43", out_key=flow, remote_ip="10.105.166.213"}
    Bridge br-ex
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        Port br-ex
            Interface br-ex
                type: internal
        Port phy-br-ex
            Interface phy-br-ex
                type: patch
                options: {peer=int-br-ex}
    ovs_version: "2.6.1"


We are able to launch the instances, but out of 5 one is failing (4:1 ratio). We have modified configuration in the below two files. Both in controller and compute node. When we run ifconfig command it is showing lot of networks and most of them are UP. We never seen these many network list earlier. I am suspecting after adding below configuration under ovs these many networks are coming. Is it a root cause 1 instance is failing out of 5?

Please let us know if you have any idea about it.





  1.  /etc/neutron/plugins/ml2/ml2_conf.ini

[ml2]
type_drivers = flat
tenant_network_types = flat
mechanism_drivers = openvswitch

[ml2_type_flat]
flat_networks = vmnet

[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
enable_security_group = True

[ovs]
bridge_mappings = vmnet:br-eno2


  1.  /opt/stack/neutron/etc/oslo-config-generator/openvswitch_agent.ini



[ovs]



bridge_mappings = vmnet:br-eno2

ovs-vsctl add-br br-int



ovs-vsctl add-br br-eno2



ovs-vsctl add-port br-eno2 eno2



From: Trinath Somanchi [mailto:trinath.somanchi at nxp.com]
Sent: Friday, December 15, 2017 10:59 PM
To: Bernd Bausch <berndbausch at gmail.com<mailto:berndbausch at gmail.com>>; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>; Ramu, MohanX <mohanx.ramu at intel.com<mailto:mohanx.ramu at intel.com>>
Cc: Sivapuram, Naga SeshendraX <naga.seshendrax.sivapuram at intel.com<mailto:naga.seshendrax.sivapuram at intel.com>>
Subject: RE: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling

Can you share the output of below command.
ovs-vsctl show.
________________________________
From: Ramu, MohanX <mohanx.ramu at intel.com<mailto:mohanx.ramu at intel.com>>
Sent: Friday, December 15, 2017 8:41:51 PM
To: Trinath Somanchi; Bernd Bausch; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Cc: Sivapuram, Naga SeshendraX
Subject: RE: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling

Hi Trinath,

We are using flat. Localrc is like local.conf which is shared already.

From: Trinath Somanchi [mailto:trinath.somanchi at nxp.com]
Sent: Friday, December 15, 2017 6:40 PM
To: Ramu, MohanX <mohanx.ramu at intel.com<mailto:mohanx.ramu at intel.com>>; Bernd Bausch <berndbausch at gmail.com<mailto:berndbausch at gmail.com>>; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Cc: Sivapuram, Naga SeshendraX <naga.seshendrax.sivapuram at intel.com<mailto:naga.seshendrax.sivapuram at intel.com>>
Subject: RE: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling

How did you configure the localrc of devstack?

Are you using flat/vlan/vxlan networking with devstack ?

For our flat network based OCATA release openstack setup, where today I fixed this issue - is working fine.

So I shared the fix, so that you can also have neutron-ovs-agent working.

In my OCATA openstack setup, I used neutron-ovs-agent and got the same below mention issue.

/
  Trinath Somanchi,
   NXP SDN/NFV Team.

From: Ramu, MohanX [mailto:mohanx.ramu at intel.com]
Sent: Friday, December 15, 2017 6:34 PM
To: Trinath Somanchi <trinath.somanchi at nxp.com<mailto:trinath.somanchi at nxp.com>>; Bernd Bausch <berndbausch at gmail.com<mailto:berndbausch at gmail.com>>; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Cc: Sivapuram, Naga SeshendraX <naga.seshendrax.sivapuram at intel.com<mailto:naga.seshendrax.sivapuram at intel.com>>
Subject: RE: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling

Hi Trinath,

We don't have /etc/neutron/plugins/ml2/openvswitch_agent.ini file. We use basic devstack, we don't have any addition into it. Please suggest us what else we should do configuration when we use  Devstack set up. We thought that Devstack itself it has all the required configurations. No much idea about Devstack.

But we had another set ups' OpenStack PIKE, OCATA and MITAKA(without devstack), there we have used LinuxBridge agent. It is working perfect.

Thanks & Regards,
Mohan Ramu

From: Trinath Somanchi [mailto:trinath.somanchi at nxp.com]
Sent: Friday, December 15, 2017 5:29 PM
To: Ramu, MohanX <mohanx.ramu at intel.com<mailto:mohanx.ramu at intel.com>>; Bernd Bausch <berndbausch at gmail.com<mailto:berndbausch at gmail.com>>; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Cc: Sivapuram, Naga SeshendraX <naga.seshendrax.sivapuram at intel.com<mailto:naga.seshendrax.sivapuram at intel.com>>
Subject: RE: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling

What is the of_listen_port configured in [ovs] at /etc/neutron/plugins/ml2/openvswitch_agent.ini (or agent config file).

The error, 2017-12-14 21:29:36.326 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout

Is caused to bridge configuration issues.

Try, of_listen_port = 7733 and spawn the VMs.

Hope this helps,

/
  Trinath Somanchi,
   NXP SDN/NFV Team.

From: Ramu, MohanX [mailto:mohanx.ramu at intel.com]
Sent: Friday, December 15, 2017 4:58 PM
To: Bernd Bausch <berndbausch at gmail.com<mailto:berndbausch at gmail.com>>; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Cc: Sivapuram, Naga SeshendraX <naga.seshendrax.sivapuram at intel.com<mailto:naga.seshendrax.sivapuram at intel.com>>
Subject: Re: [Openstack] Devstack - : Failed to allocate the network(s), not rescheduling

Hi Bernd,

PFA the local.conf files.

Below are the details used for our devstack environment.

OpenStack version : 15.0.9
Network Used : public

We are not able to launch instances every time. Sometimes instances are launching without any issue. Below are the screenshots for launching multiple instances same time.

Launched 5 instances same time, only 2 instances are created.


[cid:image001.jpg at 01D37840.174BF3D0]

Tried 10 instances same time 5 instances are created.


[cid:image002.jpg at 01D37840.174BF3D0]


Please find logs both in controller and compute side.


q-agt.log (Controller Side) :-

2017-12-14 22:39:32.880 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:37.884 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:37.963 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:42.861 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:47.865 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:47.965 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:48.074 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-12-14 22:39:48.075 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 4 from (pid=6178) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 22:39:48.076 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): DbGetCommand(column=datapath_id, table=Bridge, record=br-int) from (pid=6178) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:100
2017-12-14 22:39:48.077 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction caused no change from (pid=6178) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:124
2017-12-14 22:39:48.078 INFO neutron.plugins.ml2.drive

q-agt.log (Compute Side) :-

2017-12-14 21:29:38.212 DEBUG neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [req-dadb9cbf-741e-4378-bfc3-9dabf5e78a7d None None] ofctl request version=0x4,msg_type=0x12,msg_len=0x38,xid=0x86ee6f60,OFPFlowStatsRequest(cookie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967295,out_port=4294967295,table_id=23,type=1) result [OFPFlowStatsReply(body=[OFPFlowStats(byte_count=0,cookie=9413309945793027316L,duration_nsec=913000000,duration_sec=162395,flags=0,hard_timeout=0,idle_timeout=0,instructions=[],length=56,match=OFPMatch(oxm_fields={}),packet_count=0,priority=0,table_id=23)],flags=0,type=1)] from (pid=517) _send_msg /opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/native/o2017-12-14 21:29:31.874 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=7063) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:36.090 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=7063) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:36.875 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=7063) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:31.874 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=3621) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:36.090 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=3621) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:36.326 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-12-14 21:29:36.326 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 4 from (pid=3621) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:36.327 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): DbGetCommand(column=datapath_id, table=Bridge, record=br-int) from (pid=3621) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:100
2017-12-14 21:29:36.327 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction caused no change from (pid=3621) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:127
2017-12-14 21:29:36.327 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from de78d015444e to 0000de78d015444e
2017-12-14 21:29:36.875 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=3621) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:31.873 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=815) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:31.875 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=815) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:34.346 ERROR neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-] Switch connection timeout
2017-12-14 21:29:34.346 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 4 from (pid=815) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202
2017-12-14 21:29:34.346 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn command(idx=0): DbGetCommand(column=datapath_id, table=Bridge, record=br-int) from (pid=815) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:100
2017-12-14 21:29:34.346 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction caused no change from (pid=815) do_commit /opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:127
2017-12-14 21:29:34.347 INFO neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-] Bridge br-int changed its datapath-ID from de78d015444e to 0000de78d015444e
2017-12-14 21:29:36.875 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN] on fd 3 from (pid=815) __log_wakeup /usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

n-cpu.log (Compute Side) :-

2017-12-14 21:24:03.442 DEBUG keystoneauth1.identity.base [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mself.auth_ref ---><keystoneauth1.access.access.AccessInfoV3 object at 0x7ff9d42a2ed0>^[[00m ^[[00;33mfrom (pid=7061) get_access /usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py:138^[[00m^M
2017-12-14 21:24:03.442 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mquery_params  --> {}^[[00m^M
2017-12-14 21:24:03.443 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35murl -----::-> http://10.105.166.213/placement/resource_providers/dcf28911-6125-4bc4-87b6-bc71b913d8b2/inventories^[[00m^M<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.105.166.213%2Fplacement%2Fresource_providers%2Fdcf28911-6125-4bc4-87b6-bc71b913d8b2%2Finventories%255e%255b%255b00m%255eM&data=02%7C01%7Ctrinath.somanchi%40nxp.com%7C4cb98e3055544767b83a08d543b021c7%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636489346125753805&sdata=DmzSzlumR5kY0JmoLYS2ghpPSauqJFmVJ7OhXIFfkvU%3D&reserved=0>
2017-12-14 21:24:03.443 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mBefore connection_params^[[00m^M
2017-12-14 21:24:03.443 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mAfter connection_params --> {}^[[00m^M
2017-12-14 21:24:03.443 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mBefore send(**kwargs) ^[[00m^M
2017-12-14 21:24:03.443 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m_send_request call^[[00m^M
2017-12-14 21:24:03.520 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m_send_request ---> <Response [200]>^[[00m^M
2017-12-14 21:24:03.520 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mresp.status_code:::  --> (u'http://10.105.166.213/placement/resource_providers/dcf28911-6125-4bc4-87b6-bc71b913d8b2/inventories', 200)^[[00m^M
2017-12-14 21:24:03.520 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mENd of method^[[00m^M
2017-12-14 21:24:03.521 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mresp   ---> <Response [200]>^[[00m^M
2017-12-14 21:24:03.521 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mresp.status_code  -->(200, u'http://10.105.166.213/placement/resource_providers/dcf28911-6125-4bc4-87b6-bc71b913d8b2/inventories') ^[[00m^M
2017-12-14 21:24:03.521 DEBUG nova.scheduler.client.report [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mUpdating our resource provider generation from 31 to 32^[[00m ^[[00;33mfrom (pid=7061) _get_inventory_and_update_provider_generation /opt/stack/nova/nova/scheduler/client/report.py:449^[[00m^M
2017-12-14 21:24:03.521 DEBUG oslo_concurrency.lockutils [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mLock "compute_resources" released by "nova.compute.resource_tracker.abort_instance_claim" :: held 0.567s^[[00m ^[[00;33mfrom (pid=7061) inner /usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282^[[00m^M
2017-12-14 21:24:03.521 ERROR nova.compute.manager [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] Failed to allocate network(s)^[[00m^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00mTraceback (most recent call last):^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m  File "/opt/stack/nova/nova/compute/manager.py", line 1938, in _build_and_run_instance^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m    with self._build_resources(context, instance,^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 2756, in spawn^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m    post_xml_callback=gen_confdrive,^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m  File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5195, in _create_domain_and_network^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m    instance=instance)^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00mVirtualInterfaceCreateException: Virtual Interface creation failed^M
2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m^M
2017-12-14 21:24:03.522 DEBUG nova.compute.utils [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] Virtual Interface creation failed^[[00m ^[[00;33mfrom (pid=7061) notify_about_instance_usage /opt/stack/nova/nova/compute/utils.py:317^[[00m^M
2017-12-14 21:24:03.522 DEBUG oslo_messaging._drivers.amqpdriver [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mCAST unique_id: 9887810c2974467badeac9be672b15d2 NOTIFY exchange 'nova' topic 'notifications.error'^[[00m ^[[00;33mfrom (pid=7061) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:546^[[00m^M
2017-12-14 21:24:03.525 DEBUG oslo_messaging._drivers.amqpdriver [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mCAST unique_id: a59e16bd5ee3492c9db22dd99bd067a3 NOTIFY exchange 'nova' topic 'versioned_notifications.error'^[[00m ^[[00;33mfrom (pid=7061) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:546^[[00m^M
2017-12-14 21:24:03.526 ERROR nova.compute.manager [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] Build of instance 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd aborted: Failed to allocate the network(s), not rescheduling.^[[00m^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00mTraceback (most recent call last):^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m  File "/opt/stack/nova/nova/compute/manager.py", line 1788, in _do_build_and_run_instance^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m    return build_results.FAILED^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m  File "/opt/stack/nova/nova/compute/manager.py", line 1998, in _build_and_run_instance^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m    msg = _('Failed to allocate the network(s) with error %s, '^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00mBuildAbortException: Build of instance 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd aborted: Failed to allocate the network(s), not rescheduling.^M
2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m^M
2017-12-14 21:24:03.527 DEBUG nova.compute.manager [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] Deallocating network for instance^[[00m ^[[00;33mfrom (pid=7061) _deallocate_network /opt/stack/nova/nova/compute/manager.py:1669^[[00m^M
2017-12-14 21:24:03.527 DEBUG nova.network.neutronv2.api [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] deallocate_for_instance()^[[00m ^[[00;33mfrom (pid=7061) deallocate_for_instance /opt/stack/nova/nova/network/neutronv2/api.py:1173^[[00m^M
2017-12-14 21:24:03.527 DEBUG keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mauth_headers ----> {'X-Auth-Token': u'gAAAAABaM1TN1hhnQrnfQVASXfFXzyOOq1D8zHS9e1v54TjmkZQiXOSKBcS6kYWTVpDeRKlMHh2GZgOFQ9wJ_gSqvOQ371bTfCk61XV7PR5YeOF9pUP6VQI6x0EojazONSNJJxydImbv1Qf6FISU4HP527itUt_3cpnRnn8-1XE9M6rq_47tIN7fTwwWtkEj6C2x6Djki3Ng'}^[[00m ^[[00;33mfrom (pid=7061) request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:492^[[00m^M
2017-12-14 21:24:03.528 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mquery_params  --> {}^[[00m^M
2017-12-14 21:24:03.528 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35murl -----::-> http://10.105.166.213:9696/v2.0/ports.json?device_id=2398fce6-6f98-406d-9b1a-8cf9a1fbfccd^[[00m^M<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.105.166.213%3A9696%2Fv2.0%2Fports.json%3Fdevice_id%3D2398fce6-6f98-406d-9b1a-8cf9a1fbfccd%255e%255b%255b00m%255eM&data=02%7C01%7Ctrinath.somanchi%40nxp.com%7C4cb98e3055544767b83a08d543b021c7%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636489346125753805&sdata=5UJFiEfEtS3pHl3qotjriw2rLPhqTVaQxsswIasW5Jw%3D&reserved=0>
2017-12-14 21:24:03.528 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mBefore connection_params^[[00m^M
2017-12-14 21:24:03.528 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mAfter connection_params --> {}^[[00m^M
2017-12-14 21:24:03.528 WARNING keystoneauth1.session [^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin] ^[[01;35mBefore send(**kwargs) ^[[00m^M


From: Bernd Bausch [mailto:berndbausch at gmail.com]
Sent: Friday, December 15, 2017 6:21 AM
To: Ramu, MohanX <mohanx.ramu at intel.com<mailto:mohanx.ramu at intel.com>>; openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>
Cc: 'Jay Pipes' <jaypipes at gmail.com<mailto:jaypipes at gmail.com>>
Subject: RE: Devstack - : Failed to allocate the network(s), not rescheduling

This just says that Nova was unable to set up the network plumbing on the compute node.

Depending on the OpenStack version you deployed, you should have other messages in the same log file, for example if you are using Pike, "No more network or fixed IP to be allocated" or "Failed to allocate network(s)" (without the article). It would be helpful to know if there are such messages.

Most probably, there is more information in the Neutron logs. Since you use DevStack, you probably have an all-in-one deployment, but in case you have separate compute nodes, also check the Neutron agent log there.

You may also want to provide more information such as: What OpenStack version you have, which log are you sharing, when does this happen, does it always happen or only under certain circumstances, have you ever been able to launch an instance, what networks do you have, what network do you connect the instance to, what's your local.conf etc.

From: Ramu, MohanX [mailto:mohanx.ramu at intel.com]
Sent: Thursday, December 14, 2017 10:54 PM
To: 'openstack at lists.openstack.org' <openstack at lists.openstack.org<mailto:openstack at lists.openstack.org>>
Cc: 'Bernd Bausch' <berndbausch at gmail.com<mailto:berndbausch at gmail.com>>; Jay Pipes <jaypipes at gmail.com<mailto:jaypipes at gmail.com>>
Subject: Devstack - : Failed to allocate the network(s), not rescheduling


Hi All,

2017-12-14 02:56:06.230 DEBUG oslo_messaging._drivers.amqpdriver [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35mCAST unique_id: fa9e604328a445f39702e0089de0248a NOTIFY exchange 'nova' topic 'notifications.error'^[[00m ^[[00;33mfrom (pid=7061) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:546^[[00m^M
2017-12-14 02:56:06.236 DEBUG oslo_messaging._drivers.amqpdriver [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35mCAST unique_id: 999c7e8ac35f4b9db809312ddccc8b7f NOTIFY exchange 'nova' topic 'versioned_notifications.error'^[[00m ^[[00;33mfrom (pid=7061) _send /usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py:546^[[00m^M
2017-12-14 02:56:06.238 ERROR nova.compute.manager [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] Build of instance 8ed6476c-8c0c-4baf-b24f-8530feb836e7 aborted: Failed to allocate the network(s), not rescheduling.^[[00m^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00mTraceback (most recent call last):^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m  File "/opt/stack/nova/nova/compute/manager.py", line 1788, in _do_build_and_run_instance^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m    return build_results.FAILED^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m  File "/opt/stack/nova/nova/compute/manager.py", line 1998, in _build_and_run_instance^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m    msg = _('Failed to allocate the network(s) with error %s, '^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00mBuildAbortException: Build of instance 8ed6476c-8c0c-4baf-b24f-8530feb836e7 aborted: Failed to allocate the network(s), not rescheduling.^M
2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m^M
2017-12-14 02:56:06.239 DEBUG nova.compute.manager [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] Deallocating network for instance^[[00m ^[[00;33mfrom (pid=7061) _deallocate_network /opt/stack/nova/nova/compute/manager.py:1669^[[00m^M
2017-12-14 02:56:06.239 DEBUG nova.network.neutronv2.api [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] deallocate_for_instance()^[[00m ^[[00;33mfrom (pid=7061) deallocate_for_instance /opt/stack/nova/nova/network/neutronv2/api.py:1173^[[00m^M
2017-12-14 02:56:06.240 DEBUG keystoneauth1.session [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35mauth_headers ----> {'X-Auth-Token': u'gAAAAABaMlbpaJ9OocE4phmMPWC6eSmajqtx1id547QxiGEhxMlSddZRexD8TPPH4Il0n3mAemxAWZVXXJeMDAb47ZIlf4YNogzUitsDgkR3-VRSNbB0MdpBFv3aO_aUv5ZBOe1__Ya_bJtW33-QglQbeBqkUoUaQli8aMldZ_BhLBOsqFCYSIWXnd15z73p1j9hmcbxU3Mb'}^[[00m ^[[00;33mfrom (pid=7061) request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:492^[[00m^M
2017-12-14 02:56:06.240 WARNING keystoneauth1.session [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35mquery_params  --> {}^[[00m^M
2017-12-14 02:56:06.240 WARNING keystoneauth1.session [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35murl -----::-> http://10.105.166.213:9696/v2.0/ports.json?device_id=8ed6476c-8c0c-4baf-b24f-8530feb836e7^[[00m^M<https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2F10.105.166.213%3A9696%2Fv2.0%2Fports.json%3Fdevice_id%3D8ed6476c-8c0c-4baf-b24f-8530feb836e7%255e%255b%255b00m%255eM&data=02%7C01%7Ctrinath.somanchi%40nxp.com%7C4cb98e3055544767b83a08d543b021c7%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C636489346125753805&sdata=Ab4DFk8Tfv2p2vN8p0Cm3gaJtydUCNrt8w08Y20HZZQ%3D&reserved=0>
2017-12-14 02:56:06.241 WARNING keystoneauth1.session [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35mBefore connection_params^[[00m^M
2017-12-14 02:56:06.241 WARNING keystoneauth1.session [^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin] ^[[01;35mAfter connection_params --> {}^[[00m^M


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171219/ec92a8c5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 30036 bytes
Desc: image001.jpg
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171219/ec92a8c5/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 63974 bytes
Desc: image002.jpg
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171219/ec92a8c5/attachment-0001.jpg>


More information about the Openstack mailing list