[Openstack] Devstack - : Failed to allocate the network(s), not rescheduling

Bernd Bausch berndbausch at gmail.com
Fri Dec 15 12:10:06 UTC 2017 

Switch connection timeout points to this bug:
https://bugs.launchpad.net/neutron/+bug/1611237. It was solved a year ago in
Neutron (Nov 23, 2016), and it seems to appear when the Openvswitch agent is
started, so I am not sure if you are seeing the same thing.

 

In any case, I think it's worthwhile investigating where this timeout may
come from.

 

Bernd.

 

From: Ramu, MohanX [mailto:mohanx.ramu at intel.com] 
Sent: Friday, December 15, 2017 8:28 PM
To: Bernd Bausch <berndbausch at gmail.com>; openstack at lists.openstack.org
Cc: 'Jay Pipes' <jaypipes at gmail.com>; Sivapuram, Naga SeshendraX
<naga.seshendrax.sivapuram at intel.com>
Subject: RE: Devstack - : Failed to allocate the network(s), not
rescheduling

 

Hi Bernd,

 

PFA the local.conf files.

 

Below are the details used for our devstack environment.

 

OpenStack version : 15.0.9

Network Used : public

 

We are not able to launch instances every time. Sometimes instances are
launching without any issue. Below are the screenshots for launching
multiple instances same time. 

 

Launched 5 instances same time, only 2 instances are created.

 

 



 

Tried 10 instances same time 5 instances are created.

 

 



 

 

Please find logs both in controller and compute side. 

 

 

q-agt.log (Controller Side) :-

 

2017-12-14 22:39:32.880 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=6178) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 22:39:37.884 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=6178) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 22:39:37.963 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=6178) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 22:39:42.861 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=6178) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 22:39:47.865 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=6178) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 22:39:47.965 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=6178) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 22:39:48.074 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-]
Switch connection timeout

2017-12-14 22:39:48.075 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 4 from (pid=6178) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 22:39:48.076 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn
command(idx=0): DbGetCommand(column=datapath_id, table=Bridge,
record=br-int) from (pid=6178) do_commit
/opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:100

2017-12-14 22:39:48.077 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction
caused no change from (pid=6178) do_commit
/opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:124

2017-12-14 22:39:48.078 INFO neutron.plugins.ml2.drive

 

q-agt.log (Compute Side) :-

 

2017-12-14 21:29:38.212 DEBUG
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch
[req-dadb9cbf-741e-4378-bfc3-9dabf5e78a7d None None] ofctl request
version=0x4,msg_type=0x12,msg_len=0x38,xid=0x86ee6f60,OFPFlowStatsRequest(co
okie=0,cookie_mask=0,flags=0,match=OFPMatch(oxm_fields={}),out_group=4294967
295,out_port=4294967295,table_id=23,type=1) result
[OFPFlowStatsReply(body=[OFPFlowStats(byte_count=0,cookie=941330994579302731
6L,duration_nsec=913000000,duration_sec=162395,flags=0,hard_timeout=0,idle_t
imeout=0,instructions=[],length=56,match=OFPMatch(oxm_fields={}),packet_coun
t=0,priority=0,table_id=23)],flags=0,type=1)] from (pid=517) _send_msg
/opt/stack/neutron/neutron/plugins/ml2/drivers/openvswitch/agent/openflow/na
tive/o2017-12-14 21:29:31.874 DEBUG neutron.agent.ovsdb.native.vlog [-]
[POLLIN] on fd 3 from (pid=7063) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 21:29:36.090 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=7063) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 21:29:36.875 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=7063) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 21:29:31.874 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=3621) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 21:29:36.090 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=3621) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 21:29:36.326 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-]
Switch connection timeout

2017-12-14 21:29:36.326 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 4 from (pid=3621) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 21:29:36.327 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn
command(idx=0): DbGetCommand(column=datapath_id, table=Bridge,
record=br-int) from (pid=3621) do_commit
/opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:100

2017-12-14 21:29:36.327 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction
caused no change from (pid=3621) do_commit
/opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:127

2017-12-14 21:29:36.327 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-]
Bridge br-int changed its datapath-ID from de78d015444e to 0000de78d015444e

2017-12-14 21:29:36.875 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=3621) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 21:29:31.873 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=815) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 21:29:31.875 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=815) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 21:29:34.346 ERROR
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ofswitch [-]
Switch connection timeout

2017-12-14 21:29:34.346 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 4 from (pid=815) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

2017-12-14 21:29:34.346 DEBUG neutron.agent.ovsdb.impl_idl [-] Running txn
command(idx=0): DbGetCommand(column=datapath_id, table=Bridge,
record=br-int) from (pid=815) do_commit
/opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:100

2017-12-14 21:29:34.346 DEBUG neutron.agent.ovsdb.impl_idl [-] Transaction
caused no change from (pid=815) do_commit
/opt/stack/neutron/neutron/agent/ovsdb/impl_idl.py:127

2017-12-14 21:29:34.347 INFO
neutron.plugins.ml2.drivers.openvswitch.agent.openflow.native.ovs_bridge [-]
Bridge br-int changed its datapath-ID from de78d015444e to 0000de78d015444e

2017-12-14 21:29:36.875 DEBUG neutron.agent.ovsdb.native.vlog [-] [POLLIN]
on fd 3 from (pid=815) __log_wakeup
/usr/local/lib/python2.7/dist-packages/ovs/poller.py:202

 

n-cpu.log (Compute Side) :-

 

2017-12-14 21:24:03.442 DEBUG keystoneauth1.identity.base
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mself.auth_ref ---><keystoneauth1.access.access.AccessInfoV3 object
at 0x7ff9d42a2ed0>^[[00m ^[[00;33mfrom (pid=7061) get_access
/usr/local/lib/python2.7/dist-packages/keystoneauth1/identity/base.py:138^[[
00m^M

2017-12-14 21:24:03.442 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mquery_params  --> {}^[[00m^M

2017-12-14 21:24:03.443 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35murl -----::->
http://10.105.166.213/placement/resource_providers/dcf28911-6125-4bc4-87b6-b
c71b913d8b2/inventories^[[00m^M
<http://10.105.166.213/placement/resource_providers/dcf28911-6125-4bc4-87b6-
bc71b913d8b2/inventories%5e%5b%5b00m%5eM> 

2017-12-14 21:24:03.443 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mBefore connection_params^[[00m^M

2017-12-14 21:24:03.443 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mAfter connection_params --> {}^[[00m^M

2017-12-14 21:24:03.443 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mBefore send(**kwargs) ^[[00m^M

2017-12-14 21:24:03.443 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35m_send_request call^[[00m^M

2017-12-14 21:24:03.520 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35m_send_request ---> <Response [200]>^[[00m^M

2017-12-14 21:24:03.520 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mresp.status_code:::  -->
(u'http://10.105.166.213/placement/resource_providers/dcf28911-6125-4bc4-87b
6-bc71b913d8b2/inventories', 200)^[[00m^M

2017-12-14 21:24:03.520 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mENd of method^[[00m^M

2017-12-14 21:24:03.521 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mresp   ---> <Response [200]>^[[00m^M

2017-12-14 21:24:03.521 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mresp.status_code  -->(200,
u'http://10.105.166.213/placement/resource_providers/dcf28911-6125-4bc4-87b6
-bc71b913d8b2/inventories') ^[[00m^M

2017-12-14 21:24:03.521 DEBUG nova.scheduler.client.report
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mUpdating our resource provider generation from 31 to 32^[[00m
^[[00;33mfrom (pid=7061) _get_inventory_and_update_provider_generation
/opt/stack/nova/nova/scheduler/client/report.py:449^[[00m^M

2017-12-14 21:24:03.521 DEBUG oslo_concurrency.lockutils
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mLock "compute_resources" released by
"nova.compute.resource_tracker.abort_instance_claim" :: held 0.567s^[[00m
^[[00;33mfrom (pid=7061) inner
/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py:282^[[0
0m^M

2017-12-14 21:24:03.521 ERROR nova.compute.manager
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] Failed to allocate
network(s)^[[00m^M

2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00mTraceback (most recent call
last):^M

2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m  File
"/opt/stack/nova/nova/compute/manager.py", line 1938, in
_build_and_run_instance^M

2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m    with
self._build_resources(context, instance,^M

2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m  File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 2756, in spawn^M

2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m
post_xml_callback=gen_confdrive,^M

2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m  File
"/opt/stack/nova/nova/virt/libvirt/driver.py", line 5195, in
_create_domain_and_network^M

2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m    instance=instance)^M

2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00mVirtualInterfaceCreateException:
Virtual Interface creation failed^M

2017-12-14 21:24:03.521 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m^M

2017-12-14 21:24:03.522 DEBUG nova.compute.utils
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] Virtual Interface
creation failed^[[00m ^[[00;33mfrom (pid=7061) notify_about_instance_usage
/opt/stack/nova/nova/compute/utils.py:317^[[00m^M

2017-12-14 21:24:03.522 DEBUG oslo_messaging._drivers.amqpdriver
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mCAST unique_id: 9887810c2974467badeac9be672b15d2 NOTIFY exchange
'nova' topic 'notifications.error'^[[00m ^[[00;33mfrom (pid=7061) _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py
:546^[[00m^M

2017-12-14 21:24:03.525 DEBUG oslo_messaging._drivers.amqpdriver
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mCAST unique_id: a59e16bd5ee3492c9db22dd99bd067a3 NOTIFY exchange
'nova' topic 'versioned_notifications.error'^[[00m ^[[00;33mfrom (pid=7061)
_send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py
:546^[[00m^M

2017-12-14 21:24:03.526 ERROR nova.compute.manager
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] Build of instance
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd aborted: Failed to allocate the
network(s), not rescheduling.^[[00m^M

2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00mTraceback (most recent call
last):^M

2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m  File
"/opt/stack/nova/nova/compute/manager.py", line 1788, in
_do_build_and_run_instance^M

2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m    return
build_results.FAILED^M

2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m  File
"/opt/stack/nova/nova/compute/manager.py", line 1998, in
_build_and_run_instance^M

2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m    msg = _('Failed to allocate
the network(s) with error %s, '^M

2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00mBuildAbortException: Build of
instance 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd aborted: Failed to allocate
the network(s), not rescheduling.^M

2017-12-14 21:24:03.526 TRACE nova.compute.manager ^[[01;35m[instance:
2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] ^[[00m^M

2017-12-14 21:24:03.527 DEBUG nova.compute.manager
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd] Deallocating
network for instance^[[00m ^[[00;33mfrom (pid=7061) _deallocate_network
/opt/stack/nova/nova/compute/manager.py:1669^[[00m^M

2017-12-14 21:24:03.527 DEBUG nova.network.neutronv2.api
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35m[instance: 2398fce6-6f98-406d-9b1a-8cf9a1fbfccd]
deallocate_for_instance()^[[00m ^[[00;33mfrom (pid=7061)
deallocate_for_instance
/opt/stack/nova/nova/network/neutronv2/api.py:1173^[[00m^M

2017-12-14 21:24:03.527 DEBUG keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mauth_headers ----> {'X-Auth-Token':
u'gAAAAABaM1TN1hhnQrnfQVASXfFXzyOOq1D8zHS9e1v54TjmkZQiXOSKBcS6kYWTVpDeRKlMHh
2GZgOFQ9wJ_gSqvOQ371bTfCk61XV7PR5YeOF9pUP6VQI6x0EojazONSNJJxydImbv1Qf6FISU4H
P527itUt_3cpnRnn8-1XE9M6rq_47tIN7fTwwWtkEj6C2x6Djki3Ng'}^[[00m ^[[00;33mfrom
(pid=7061) request
/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:492^[[00m^M

2017-12-14 21:24:03.528 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mquery_params  --> {}^[[00m^M

2017-12-14 21:24:03.528 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35murl -----::->
http://10.105.166.213:9696/v2.0/ports.json?device_id=2398fce6-6f98-406d-9b1a
-8cf9a1fbfccd^[[00m^M
<http://10.105.166.213:9696/v2.0/ports.json?device_id=2398fce6-6f98-406d-9b1
a-8cf9a1fbfccd%5e%5b%5b00m%5eM> 

2017-12-14 21:24:03.528 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mBefore connection_params^[[00m^M

2017-12-14 21:24:03.528 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mAfter connection_params --> {}^[[00m^M

2017-12-14 21:24:03.528 WARNING keystoneauth1.session
[^[[01;36mreq-2e8bd076-f1a4-45ff-a7e7-f54b8bee9c96 ^[[00;36madmin admin]
^[[01;35mBefore send(**kwargs) ^[[00m^M

 

 

From: Bernd Bausch [mailto:berndbausch at gmail.com] 
Sent: Friday, December 15, 2017 6:21 AM
To: Ramu, MohanX <mohanx.ramu at intel.com <mailto:mohanx.ramu at intel.com> >;
openstack at lists.openstack.org <mailto:openstack at lists.openstack.org> 
Cc: 'Jay Pipes' <jaypipes at gmail.com <mailto:jaypipes at gmail.com> >
Subject: RE: Devstack - : Failed to allocate the network(s), not
rescheduling

 

This just says that Nova was unable to set up the network plumbing on the
compute node. 

 

Depending on the OpenStack version you deployed, you should have other
messages in the same log file, for example if you are using Pike, "No more
network or fixed IP to be allocated" or "Failed to allocate network(s)"
(without the article). It would be helpful to know if there are such
messages. 

 

Most probably, there is more information in the Neutron logs. Since you use
DevStack, you probably have an all-in-one deployment, but in case you have
separate compute nodes, also check the Neutron agent log there.

 

You may also want to provide more information such as: What OpenStack
version you have, which log are you sharing, when does this happen, does it
always happen or only under certain circumstances, have you ever been able
to launch an instance, what networks do you have, what network do you
connect the instance to, what's your local.conf etc.

 

From: Ramu, MohanX [mailto:mohanx.ramu at intel.com] 
Sent: Thursday, December 14, 2017 10:54 PM
To: 'openstack at lists.openstack.org' <openstack at lists.openstack.org
<mailto:openstack at lists.openstack.org> >
Cc: 'Bernd Bausch' <berndbausch at gmail.com <mailto:berndbausch at gmail.com> >;
Jay Pipes <jaypipes at gmail.com <mailto:jaypipes at gmail.com> >
Subject: Devstack - : Failed to allocate the network(s), not rescheduling

 

 

Hi All,

 

2017-12-14 02:56:06.230 DEBUG oslo_messaging._drivers.amqpdriver
[^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin]
^[[01;35mCAST unique_id: fa9e604328a445f39702e0089de0248a NOTIFY exchange
'nova' topic 'notifications.error'^[[00m ^[[00;33mfrom (pid=7061) _send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py
:546^[[00m^M

2017-12-14 02:56:06.236 DEBUG oslo_messaging._drivers.amqpdriver
[^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin]
^[[01;35mCAST unique_id: 999c7e8ac35f4b9db809312ddccc8b7f NOTIFY exchange
'nova' topic 'versioned_notifications.error'^[[00m ^[[00;33mfrom (pid=7061)
_send
/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py
:546^[[00m^M

2017-12-14 02:56:06.238 ERROR nova.compute.manager
[^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin]
^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] Build of instance
8ed6476c-8c0c-4baf-b24f-8530feb836e7 aborted: Failed to allocate the
network(s), not rescheduling.^[[00m^M

2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance:
8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00mTraceback (most recent call
last):^M

2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance:
8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m  File
"/opt/stack/nova/nova/compute/manager.py", line 1788, in
_do_build_and_run_instance^M

2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance:
8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m    return
build_results.FAILED^M

2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance:
8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m  File
"/opt/stack/nova/nova/compute/manager.py", line 1998, in
_build_and_run_instance^M

2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance:
8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m    msg = _('Failed to allocate
the network(s) with error %s, '^M

2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance:
8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00mBuildAbortException: Build of
instance 8ed6476c-8c0c-4baf-b24f-8530feb836e7 aborted: Failed to allocate
the network(s), not rescheduling.^M

2017-12-14 02:56:06.238 TRACE nova.compute.manager ^[[01;35m[instance:
8ed6476c-8c0c-4baf-b24f-8530feb836e7] ^[[00m^M

2017-12-14 02:56:06.239 DEBUG nova.compute.manager
[^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin]
^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7] Deallocating
network for instance^[[00m ^[[00;33mfrom (pid=7061) _deallocate_network
/opt/stack/nova/nova/compute/manager.py:1669^[[00m^M

2017-12-14 02:56:06.239 DEBUG nova.network.neutronv2.api
[^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin]
^[[01;35m[instance: 8ed6476c-8c0c-4baf-b24f-8530feb836e7]
deallocate_for_instance()^[[00m ^[[00;33mfrom (pid=7061)
deallocate_for_instance
/opt/stack/nova/nova/network/neutronv2/api.py:1173^[[00m^M

2017-12-14 02:56:06.240 DEBUG keystoneauth1.session
[^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin]
^[[01;35mauth_headers ----> {'X-Auth-Token':
u'gAAAAABaMlbpaJ9OocE4phmMPWC6eSmajqtx1id547QxiGEhxMlSddZRexD8TPPH4Il0n3mAem
xAWZVXXJeMDAb47ZIlf4YNogzUitsDgkR3-VRSNbB0MdpBFv3aO_aUv5ZBOe1__Ya_bJtW33-Qgl
QbeBqkUoUaQli8aMldZ_BhLBOsqFCYSIWXnd15z73p1j9hmcbxU3Mb'}^[[00m ^[[00;33mfrom
(pid=7061) request
/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:492^[[00m^M

2017-12-14 02:56:06.240 WARNING keystoneauth1.session
[^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin]
^[[01;35mquery_params  --> {}^[[00m^M

2017-12-14 02:56:06.240 WARNING keystoneauth1.session
[^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin]
^[[01;35murl -----::->
http://10.105.166.213:9696/v2.0/ports.json?device_id=8ed6476c-8c0c-4baf-b24f
-8530feb836e7^[[00m^M
<http://10.105.166.213:9696/v2.0/ports.json?device_id=8ed6476c-8c0c-4baf-b24
f-8530feb836e7%5e%5b%5b00m%5eM> 

2017-12-14 02:56:06.241 WARNING keystoneauth1.session
[^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin]
^[[01;35mBefore connection_params^[[00m^M

2017-12-14 02:56:06.241 WARNING keystoneauth1.session
[^[[01;36mreq-0949894a-1c03-4956-8582-49637a29e3b0 ^[[00;36madmin admin]
^[[01;35mAfter connection_params --> {}^[[00m^M

                                                

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171215/fc2b0ded/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 30036 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171215/fc2b0ded/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 63974 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20171215/fc2b0ded/attachment-0001.jpg>

More information about the Openstack mailing list