<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
All,
<div><br>
</div>
<div>I’m noticing the following Neutron server error “failed scheduling network” when a user attempts to startup multiple instances at one time. In this case, a number of instances fail to configure their network access.</div>
<div><br>
</div>
<div>I suspect this is a scaling issue, as we’re running a relatively small Openstack testbed and have multiple services running on the same controller node. Neutron is configure with a single DHCP agent. Instances are being brought up on a flat network that
maps directly to our campus backbone. A sample of the neutron server log file shows the following:</div>
<div><br>
</div>
<div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:56 WARNING [neutron.db.agentschedulers_db] Fail scheduling network {'status': u'A</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7', u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up': True, 'tenant_id': u'4414273897</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id': </div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:56 WARNING [neutron.db.agentschedulers_db] Fail scheduling network {'status': u'A</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7', u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up': True, 'tenant_id': u'4414273897</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id': </div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}</div>
</div>
<div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 WARNING [neutron.db.agentschedulers_db] Fail scheduling network {'status': u'A</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7', u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up': True, 'tenant_id': u'4414273897</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id': </div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 WARNING [neutron.db.agentschedulers_db] Fail scheduling network {'status': u'A</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7', u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up': True, 'tenant_id': u'4414273897</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id': </div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:58 WARNING [neutron.db.agentschedulers_db] Fail scheduling network {'status': u'A</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7', u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up': True, 'tenant_id': u'4414273897</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id': </div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:05:09 WARNING [neutron.db.agentschedulers_db] Fail scheduling network {'status': u'A</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
CTIVE', 'subnets': [u'4df48f6b-8d6e-4a5a-b922-d3503add14a7', u'b90d7486-7ecb-425c-b9ba-180ba4c936df'], 'name</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
': u'campus-net', 'provider:physical_network': u'default', 'admin_state_up': True, 'tenant_id': u'4414273897</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
e44c35ab99cd73a3996787', 'provider:network_type': u'flat', 'router:external': False, 'shared': False, 'id': </div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
u'a9672c56-5491-4052-aa55-c25507ba4be7', 'provider:segmentation_id': None}</div>
</div>
<div><br>
</div>
<div>etc., etc., etc.</div>
<div><br>
</div>
<div>Question. Is this a simple scaling issue, where the DHCP agent is unable to respond to scheduling requests from the neutron server? Or is my understanding naive, and something else is going on? The corresponding DHCP log for the same time period, shows
the following:</div>
<div><br>
</div>
<div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:56 DEBUG [neutron.agent.linux.utils] Running command: ['sudo', '/usr/bin/neutro</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
n-rootwrap', '/etc/neutron/rootwrap.conf', 'kill', '-HUP', '46267']</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.agent.linux.utils] </div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
Command: ['sudo', '/usr/bin/neutron-rootwrap', '/etc/neutron/rootwrap.conf', 'kill', '-HUP', '46267']</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
Exit code: 0</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
Stdout: ''</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
Stderr: ''</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.agent.linux.dhcp] Reloading allocations for network: a9672c56</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
-5491-4052-aa55-c25507ba4be7</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp] Making synchronous call on q-plugi</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
n ...</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp] MSG_ID is 6be3406bc1c747e2890cd53d</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
c24f5e4d</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp] UNIQUE_ID is db708afb7b6848be9f07e</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
395e7c27cad.</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp] Making asynchronous cast on q-plug</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
in...</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp] UNIQUE_ID is cb882dab0a6e4c12b1145</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
189f45a345f.</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [amqp] Closed channel #1</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [amqp] using channel_id: 1</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [amqp] Channel open</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [amqp] Closed channel #1</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [amqp] using channel_id: 1</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [amqp] Channel open</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp] received {u'_context_roles': [u'ad</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
min'], u'_context_read_deleted': u'no', u'_context_tenant_id': u'f59f41096eb142ccbe72e7cf6702518c', u'args':</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
{u'payload': {u'port': {u'status': u'DOWN', u'binding:host_id': u'cn6', u'name': u'', u'allowed_address_pai</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
rs': [], u'admin_state_up': True, u'network_id': u'a9672c56-5491-4052-aa55-c25507ba4be7', u'tenant_id': u'44</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
14273897e44c35ab99cd73a3996787', u'binding:vif_type': u'ovs', u'device_owner': u'compute:nova', u'binding:ca</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
pabilities': {u'port_filter': True}, u'mac_address': u'fa:16:3e:4f:88:54', u'fixed_ips': [{u'subnet_id': u'4</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
df48f6b-8d6e-4a5a-b922-d3503add14a7', u'ip_address': u'173.23.187.15'}], u'id': u'1924e8b5-ea22-4474-adc1-fc</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
3194401cc8', u'security_groups': [u'2f8ef352-adec-48f5-ba17-bb55f85de7de'], u'device_id': u'7bf73c30-d949-4f</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
34-886f-68c9a503e963'}}}, u'namespace': None, u'_unique_id': u'a7d139060a874c3b832af33491d8c469', u'_context</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
_is_admin': True, u'version': u'1.0', u'_context_project_id': u'f59f41096eb142ccbe72e7cf6702518c', u'_contex</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
t_timestamp': u'2014-07-15 19:01:54.484397', u'_context_user_id': u'6f0f1783946348f7b01881501e2f5b9c', u'met</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
hod': u'port_create_end'}</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp] unpacked context: {'user_id': u'6f</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
0f1783946348f7b01881501e2f5b9c', 'roles': [u'admin'], 'tenant_id': u'f59f41096eb142ccbe72e7cf6702518c', 'is_</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
admin': True, 'timestamp': u'2014-07-15 19:01:54.484397', 'project_id': u'f59f41096eb142ccbe72e7cf6702518c',</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
'read_deleted': u'no'}</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp] received {u'_context_roles': [u'ad</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
min'], u'_context_read_deleted': u'no', u'_context_tenant_id': u'f59f41096eb142ccbe72e7cf6702518c', u'args':</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
{u'payload': {u'port': {u'status': u'DOWN', u'binding:host_id': u'cn2', u'name': u'', u'allowed_address_pai</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
rs': [], u'admin_state_up': True, u'network_id': u'a9672c56-5491-4052-aa55-c25507ba4be7', u'tenant_id': u'44</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
14273897e44c35ab99cd73a3996787', u'binding:vif_type': u'ovs', u'device_owner': u'compute:nova', u'binding:ca</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
pabilities': {u'port_filter': True}, u'mac_address': u'fa:16:3e:4d:5b:33', u'fixed_ips': [{u'subnet_id': u'4</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
df48f6b-8d6e-4a5a-b922-d3503add14a7', u'ip_address': u'173.23.187.16'}], u'id': u'02438722-85a7-4d49-b74b-66</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
61650bbcd0', u'security_groups': [u'2f8ef352-adec-48f5-ba17-bb55f85de7de'], u'device_id': u'242437ce-d346-43</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
d6-b6c9-41777eb8d669'}}}, u'namespace': None, u'_unique_id': u'5a162a32393f4506a8f9d42b963c44b2', u'_context</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
_is_admin': True, u'version': u'1.0', u'_context_project_id': u'f59f41096eb142ccbe72e7cf6702518c', u'_contex</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
t_timestamp': u'2014-07-15 19:01:54.490335', u'_context_user_id': u'6f0f1783946348f7b01881501e2f5b9c', u'met</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
hod': u'port_create_end'}</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
NEUTRON: 2014-07-15 14:01:57 DEBUG [neutron.openstack.common.rpc.amqp] unpacked context: {'user_id': u'6f</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
0f1783946348f7b01881501e2f5b9c', 'roles': [u'admin'], 'tenant_id': u'f59f41096eb142ccbe72e7cf6702518c', 'is_</div>
<div style="margin: 0px; font-family: Courier; color: rgb(76, 47, 45); background-color: rgb(223, 219, 196);">
admin</div>
</div>
<div><br>
</div>
<div>Do we have some type of a race condition going on here? </div>
<div><br>
</div>
<div>Thoughts?</div>
<div>/ross</div>
<div><br>
</div>
<div><span style="orphans: 2; text-align: -webkit-auto; widows: 2;">--</span></div>
<div apple-content-edited="true">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; orphans: 2; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>Ross Lillie</div>
<div>Distinguished Member of Technical Staff</div>
<div>Motorola Solutions, Inc.</div>
<div><br>
</div>
<div><a href="http://motorolasolutions.com">motorolasolutions.com</a></div>
</div>
<span style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">O:
+1.847.576.0012</span>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
M: +1.847.980.2241</div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
E: <a href="mailto:ross.lillie@motorolasolutions.com">ross.lillie@motorolasolutions.com</a></div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<br>
</div>
<div class="page" title="Page 1" style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<br class="Apple-interchange-newline">
<span><img height="33" width="277" apple-inline="yes" id="ABFDA6B3-CB6E-4649-9F93-BEAB8EF8BCC6" apple-width="yes" apple-height="yes" src="cid:75FF7557-F6FD-4406-A0BB-5CBD9BE3780A@comm.mot.com"></span>
</div>
</div>
<br>
</body>
</html>