Open Stack

All,

I’m noticing the following Neutron server error “failed scheduling network” when a user attempts to startup multiple instances at one time.  In this case, a number of instances fail to configure their network access.

I suspect this is a scaling issue, as we’re running a relatively small Openstack testbed and have multiple services running on the same controller node.  Neutron is configure with a single DHCP agent.  Instances are being brought up on a flat network that maps directly to our campus backbone.  A sample of the neutron server log file shows the following:

NEUTRON: 2014-07-15 14:01:56  WARNING [neutron.db.agentschedulers_db] Fail scheduling network {'status': u'A
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7', u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up': True, 'tenant_id': u'4414273897
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}
NEUTRON: 2014-07-15 14:01:56  WARNING [neutron.db.agentschedulers_db] Fail scheduling network {'status': u'A
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7', u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up': True, 'tenant_id': u'4414273897
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}
NEUTRON: 2014-07-15 14:01:57  WARNING [neutron.db.agentschedulers_db] Fail scheduling network {'status': u'A
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7', u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up': True, 'tenant_id': u'4414273897
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}
NEUTRON: 2014-07-15 14:01:57  WARNING [neutron.db.agentschedulers_db] Fail scheduling network {'status': u'A
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7', u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up': True, 'tenant_id': u'4414273897
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}
NEUTRON: 2014-07-15 14:01:58  WARNING [neutron.db.agentschedulers_db] Fail scheduling network {'status': u'A
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7', u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up': True, 'tenant_id': u'4414273897
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}
NEUTRON: 2014-07-15 14:05:09  WARNING [neutron.db.agentschedulers_db] Fail scheduling network {'status': u'A
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7', u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up': True, 'tenant_id': u'4414273897
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id':
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}

etc., etc., etc.

Question.  Is this a simple scaling issue, where the DHCP agent is unable to respond to scheduling requests from the neutron server? Or is my understanding naive, and something else is going on?  The corresponding DHCP log for the same time period, shows the following:

NEUTRON: 2014-07-15 14:01:56    DEBUG [neutron.agent.linux.utils] Running command: ['sudo', '/usr/bin/neutro
n-rootwrap', '/etc/neutron/rootwrap.conf', 'kill', '-HUP', '46267']
NEUTRON: 2014-07-15 14:01:57    DEBUG [neutron.agent.linux.utils]
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'kill', '-HUP', '46267']
Exit code: 0
Stdout: ''
Stderr: ''
NEUTRON: 2014-07-15 14:01:57    DEBUG [neutron.agent.linux.dhcp] Reloading allocations for network: a9672c56
-5491-4052-aa55-c25507ba4be7
NEUTRON: 2014-07-15 14:01:57    DEBUG [neutron.openstack.common.rpc.amqp] Making synchronous call on q-plugi
n ...
NEUTRON: 2014-07-15 14:01:57    DEBUG [neutron.openstack.common.rpc.amqp] MSG_ID is 6be3406bc1c747e2890cd53d
c24f5e4d
NEUTRON: 2014-07-15 14:01:57    DEBUG [neutron.openstack.common.rpc.amqp] UNIQUE_ID is db708afb7b6848be9f07e
395e7c27cad.
NEUTRON: 2014-07-15 14:01:57    DEBUG [neutron.openstack.common.rpc.amqp] Making asynchronous cast on q-plug
in...
NEUTRON: 2014-07-15 14:01:57    DEBUG [neutron.openstack.common.rpc.amqp] UNIQUE_ID is cb882dab0a6e4c12b1145
189f45a345f.
NEUTRON: 2014-07-15 14:01:57    DEBUG [amqp] Closed channel #1
NEUTRON: 2014-07-15 14:01:57    DEBUG [amqp] using channel_id: 1
NEUTRON: 2014-07-15 14:01:57    DEBUG [amqp] Channel open
NEUTRON: 2014-07-15 14:01:57    DEBUG [amqp] Closed channel #1
NEUTRON: 2014-07-15 14:01:57    DEBUG [amqp] using channel_id: 1
NEUTRON: 2014-07-15 14:01:57    DEBUG [amqp] Channel open
NEUTRON: 2014-07-15 14:01:57    DEBUG [neutron.openstack.common.rpc.amqp] received {u'_context_roles': [u'ad
min'], u'_context_read_deleted': u'no', u'_context_tenant_id': u'f59f41096eb142ccbe72e7cf6702518c', u'args':
 {u'payload': {u'port': {u'status': u'DOWN', u'binding:host_id': u'cn6', u'name': u'', u'allowed_address_pai
rs': [], u'admin_state_up': True, u'network_id': u'a9672c56-5491-4052-aa55-c25507ba4be7', u'tenant_id': u'44
14273897e44c35ab99cd73a3996787', u'binding:vif_type': u'ovs', u'device_owner': u'compute:nova', u'binding:ca
pabilities': {u'port_filter': True}, u'mac_address': u'fa:16:3e:4f:88:54', u'fixed_ips': [{u'subnet_id': u'4
df48f6b-8d6e-4a5a-b922-d3503add14a7', u'ip_address': u'173.23.187.15'}], u'id': u'1924e8b5-ea22-4474-adc1-fc
3194401cc8', u'security_groups': [u'2f8ef352-adec-48f5-ba17-bb55f85de7de'], u'device_id': u'7bf73c30-d949-4f
34-886f-68c9a503e963'}}}, u'namespace': None, u'_unique_id': u'a7d139060a874c3b832af33491d8c469', u'_context
_is_admin': True, u'version': u'1.0', u'_context_project_id': u'f59f41096eb142ccbe72e7cf6702518c', u'_contex
t_timestamp': u'2014-07-15 19:01:54.484397', u'_context_user_id': u'6f0f1783946348f7b01881501e2f5b9c', u'met
hod': u'port_create_end'}
NEUTRON: 2014-07-15 14:01:57    DEBUG [neutron.openstack.common.rpc.amqp] unpacked context: {'user_id': u'6f
0f1783946348f7b01881501e2f5b9c', 'roles': [u'admin'], 'tenant_id': u'f59f41096eb142ccbe72e7cf6702518c', 'is_
admin': True, 'timestamp': u'2014-07-15 19:01:54.484397', 'project_id': u'f59f41096eb142ccbe72e7cf6702518c',
 'read_deleted': u'no'}
NEUTRON: 2014-07-15 14:01:57    DEBUG [neutron.openstack.common.rpc.amqp] received {u'_context_roles': [u'ad
min'], u'_context_read_deleted': u'no', u'_context_tenant_id': u'f59f41096eb142ccbe72e7cf6702518c', u'args':
 {u'payload': {u'port': {u'status': u'DOWN', u'binding:host_id': u'cn2', u'name': u'', u'allowed_address_pai
rs': [], u'admin_state_up': True, u'network_id': u'a9672c56-5491-4052-aa55-c25507ba4be7', u'tenant_id': u'44
14273897e44c35ab99cd73a3996787', u'binding:vif_type': u'ovs', u'device_owner': u'compute:nova', u'binding:ca
pabilities': {u'port_filter': True}, u'mac_address': u'fa:16:3e:4d:5b:33', u'fixed_ips': [{u'subnet_id': u'4
df48f6b-8d6e-4a5a-b922-d3503add14a7', u'ip_address': u'173.23.187.16'}], u'id': u'02438722-85a7-4d49-b74b-66
61650bbcd0', u'security_groups': [u'2f8ef352-adec-48f5-ba17-bb55f85de7de'], u'device_id': u'242437ce-d346-43
d6-b6c9-41777eb8d669'}}}, u'namespace': None, u'_unique_id': u'5a162a32393f4506a8f9d42b963c44b2', u'_context
_is_admin': True, u'version': u'1.0', u'_context_project_id': u'f59f41096eb142ccbe72e7cf6702518c', u'_contex
t_timestamp': u'2014-07-15 19:01:54.490335', u'_context_user_id': u'6f0f1783946348f7b01881501e2f5b9c', u'met
hod': u'port_create_end'}
NEUTRON: 2014-07-15 14:01:57    DEBUG [neutron.openstack.common.rpc.amqp] unpacked context: {'user_id': u'6f
0f1783946348f7b01881501e2f5b9c', 'roles': [u'admin'], 'tenant_id': u'f59f41096eb142ccbe72e7cf6702518c', 'is_
admin

Do we have some type of a race condition going on here?

Thoughts?
/ross

--
Ross Lillie
Distinguished Member of Technical Staff
Motorola Solutions, Inc.

motorolasolutions.com<http://motorolasolutions.com>
O: +1.847.576.0012
M: +1.847.980.2241
E: ross.lillie at motorolasolutions.com<mailto:ross.lillie at motorolasolutions.com>


[cid:75FF7557-F6FD-4406-A0BB-5CBD9BE3780A at comm.mot.com]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140715/2266f0c5/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: MSI-Email-Identity-sm.png
Type: image/png
Size: 10441 bytes
Desc: MSI-Email-Identity-sm.png
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140715/2266f0c5/attachment.png>