<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">You need to make a few changes in nova.conf but you are running liberty right? Not mitaka!<div class=""><br class=""></div><div class="">So therefore, your error maybe diff, how many nodes are you running? </div><div class=""><br class=""></div><div class="">Thanks<br class=""><div><blockquote type="cite" class=""><div class="">On Apr 27, 2016, at 16:01, The O.G. <<a href="mailto:theoriginalguru@gmail.com" class="">theoriginalguru@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class=""><div class=""><div class=""><div class=""><div class="">Remo & other OpenStack:<br class=""><br class=""></div>I've been looking into this more today and noticed this WARNING message from the nova-compute.log (compute node):<br class=""><br class="">2016-04-27 15:39:39.191 856 WARNING oslo_config.cfg [-] Option "username" from group "neutron" is deprecated. Use option "user-name" from group "neutron".<br class=""><br class=""></div>This message is written around the same time as this message in neutron server.log (controller node):<br class=""><br class="">2016-04-27 15:41:57.398 2262 ERROR neutron.notifiers.nova BadRequest: Expecting to find id or name in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-854b955c-2732-464b-b8b5-d087bf4ef9f8)<br class=""><br class=""></div>Also, noticed similar message in keystone.log (controller node):<br class=""><br class="">2016-04-27 15:46:57.167 4593 WARNING keystone.common.wsgi [req-465cfd8f-194e-47be-a83f-fe8ae90b8766 - - - - -] Expecting to find id or name in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error.<br class=""><br class=""></div>Can anyone please help me narrow down where I might be able to find the cause of these error messages?<br class=""><br class=""></div>Thanks.<br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Apr 27, 2016 at 3:56 PM, The O.G. <span dir="ltr" class=""><<a href="mailto:theoriginalguru@gmail.com" target="_blank" class="">theoriginalguru@gmail.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr" class="">Yes, this is on both controller and compute node.<div class=""><div class="h5"><br class=""><div class=""><div class="gmail_extra"><br class=""><div class="gmail_quote">On Wed, Apr 27, 2016 at 3:41 PM, Remo Mattei <span dir="ltr" class=""><<a href="mailto:remo@italy1.com" target="_blank" class="">remo@italy1.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word" class="">is this on the controller and computes?<br class=""><div class=""><blockquote type="cite" class=""><div class=""><div class=""><div class="">On Apr 27, 2016, at 15:40, The O.G. <<a href="mailto:theoriginalguru@gmail.com" target="_blank" class="">theoriginalguru@gmail.com</a>> wrote:</div><br class=""></div></div><div class=""><div dir="ltr" class=""><div class=""><div class="">To be more specific, SELinux is enabled, but in permissive mode:<br class=""><br class=""># sestatus <br class="">SELinux status: enabled<br class="">SELinuxfs mount: /sys/fs/selinux<br class="">SELinux root directory: /etc/selinux<br class="">Loaded policy name: targeted<br class="">Current mode: permissive<br class="">Mode from config file: permissive<br class="">Policy MLS status: enabled<br class="">Policy deny_unknown status: allowed<br class="">Max kernel policy version: 28<br class=""><br class=""></div></div><div class=""><div class="gmail_extra"><br class=""><div class="gmail_quote"><div class=""><div class="">On Wed, Apr 27, 2016 at 3:36 PM, The O.G. <span dir="ltr" class=""><<a href="mailto:theoriginalguru@gmail.com" target="_blank" class="">theoriginalguru@gmail.com</a>></span> wrote:<br class=""></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><br class=""><div class="gmail_extra"><br class=""><div class="gmail_quote"><div class=""><div class=""><span class="">On Wed, Apr 27, 2016 at 3:34 PM, Remo Mattei <span dir="ltr" class=""><<a href="mailto:remo@italy1.com" target="_blank" class="">remo@italy1.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class="">are you running SELinux?<div class="">sestatus on all of them and let me know. </div><div class=""><br class=""></div></div></blockquote></span><div class="">yes, SELinux enabled on all systems. are you suggesting I try with SELinux disabled? (I can do that...)<br class=""><br class=""></div><div class="">thanks...<br class=""><br class=""></div></div></div><div class=""><div class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""></div><div class="">Remo <br class=""><div class=""><blockquote type="cite" class=""><div class=""><div class=""><div class=""><div class=""><div class="">On Apr 27, 2016, at 15:23, The O.G. <<a href="mailto:theoriginalguru@gmail.com" target="_blank" class="">theoriginalguru@gmail.com</a>> wrote:</div><br class=""></div></div></div></div><div class=""><div class=""><div class=""><div dir="ltr" class="">Hi Remo,<br class=""><div class=""><div class="gmail_extra"><br class=""><div class="gmail_quote"><div class=""><div class="">On Thu, Apr 21, 2016 at 7:00 AM, Remo Mattei <span dir="ltr" class=""><<a href="mailto:remo@italy1.com" target="_blank" class="">remo@italy1.com</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class="">has this been a new installation or an upgrade?</div></blockquote><div class=""><br class=""></div><div class="">This is a brand new installation of Openstack/Liberty on CentOS7<br class=""> <br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">Not sure what you mean with option 2 since the install guide does not say anything about options. </div></div></blockquote><div class=""><br class=""></div><div class=""> If you see here:<br class=""><br class=""><a href="http://docs.openstack.org/liberty/install-guide-rdo/neutron-controller-install.html" target="_blank" class="">http://docs.openstack.org/liberty/install-guide-rdo/neutron-controller-install.html</a><br class=""><br class=""></div><div class="">Under "Configure networking options", there's an option 1 (Provider network) and option 2 (self-service network). All I'm saying is that we have been using "option 2" while following the guide.<br class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">did you check if rabbit is up?<br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">Yes. We repeated all the "verify operations" steps in the guide to be sure that the services are working. All verification steps passed. I assume this means rabbit is up or it would not have passed. The rabbit process is running also. Is there some other step you recommend to verify the proper operation of rabbitmq?<br class=""> <br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""></div><div class="">did you check and see if the services are all good</div></div></blockquote><div class=""><br class=""></div><div class="">Well, something is *not* good, hence we have problems. But, if you mean by "systemctl status <service_name>", then yes, all services appear to be good. Is there certain commands you would recommend other than systemctl status to check on services?<br class=""> <br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class=""><div class="">did you check and see if the agents are happy?</div></div></blockquote><div class=""><br class=""></div><div class=""> I don't know what "happy" means, but this is the output of neutron agent-list:<br class=""><br class=""># neutron agent-list<br class="">+--------------------------------------+--------------------+-------------------------------+-------+----------------+---------------------------+<br class="">| id | agent_type | host | alive | admin_state_up | binary |<br class="">+--------------------------------------+--------------------+-------------------------------+-------+----------------+---------------------------+<br class="">| 321c6188-0a8b-435e-98c7-42fcd2b9f9de | Linux bridge agent | controller | :-) | True | neutron-linuxbridge-agent |<br class="">| 3cc20152-3db7-4a63-8b9e-530cbf272ab3 | Linux bridge agent | compute | :-) | True | neutron-linuxbridge-agent |<br class="">| 60351fc0-873d-4fa3-b295-2146174e567c | Metadata agent | controller | :-) | True | neutron-metadata-agent |<br class="">| a2dcef04-8ccf-4034-a7f9-4558cc8798f9 | L3 agent | controller | :-) | True | neutron-l3-agent |<br class="">| d8aed254-edd0-4f6a-a264-4953a1a2c7e8 | DHCP agent | controller | :-) | True | neutron-dhcp-agent |<br class="">+--------------------------------------+--------------------+-------------------------------+-------+----------------+---------------------------+<br class=""><br class=""></div><div class="">If the smily face :-) means happy, then yes I think they are happy. Is there something else I should check?<br class=""><br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""><br class=""></div><div class="">just a few things to check let us know!</div><div class=""><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">Thanks for your reply. Do you have any hints or insight on the "BadRequest" message in the logs I posted ?<br class=""><br class=""></div><div class="">Bond<br class=""><br class=""></div></div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="word-wrap:break-word" class=""><div class=""></div><div class="">Remo <br class=""><div class=""><blockquote type="cite" class=""><div class=""><div class=""><div class=""><div class=""><div class="">On Apr 20, 2016, at 19:05, The O.G. <<a href="mailto:theoriginalguru@gmail.com" target="_blank" class="">theoriginalguru@gmail.com</a>> wrote:</div><br class=""></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class="">We are following:<br class=""><br class=""><a href="http://docs.openstack.org/liberty/install-guide-rdo/index.html" target="_blank" class="">http://docs.openstack.org/liberty/install-guide-rdo/index.html</a><br class=""><br class=""></div>Using CentOS7. We've chosen the option 2 for networking, where we have both a private and public network. We've pretty much following the guide step by step up to and finishing "Add the dashboard" step. All the verification steps passed.<br class=""><br class=""></div>Now, we are trying to launch an instance on the public and private networks as outlined here:<br class=""><br class=""><a href="http://docs.openstack.org/liberty/install-guide-rdo/launch-instance-public.html" target="_blank" class="">http://docs.openstack.org/liberty/install-guide-rdo/launch-instance-public.html</a><br class=""><br class=""><a href="http://docs.openstack.org/liberty/install-guide-rdo/launch-instance-private.html" target="_blank" class="">http://docs.openstack.org/liberty/install-guide-rdo/launch-instance-private.html</a><br class=""><br class=""></div>In both attempts to launch an instance, the instance gets aborted and shows status = Error. With this message:<br class=""><br class="">Build of instance 93bf7968-09b2-4890-9c3a-0d909ce57017 aborted: Failed to allocate the network(s), not rescheduling.<br class=""><br class=""></div>On the compute node, in /var/log/nova/nova-compute.log, we get these messages (I'm filtering on ERROR|WARNING only):<br class=""><br class="">2016-04-20 18:35:31.484 860 WARNING nova.virt.libvirt.driver [req-b5ec5b9d-2299-411f-b259-7a48b2b47d26 e11c48fa3827406c9e6304f88332e059 a8f41d79dc7a490c91013616f66c0134 - - -] [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] Timeout waiting for vif plugging callback for instance 93bf7968-09b2-4890-9c3a-0d909ce57017<br class="">2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [req-b5ec5b9d-2299-411f-b259-7a48b2b47d26 e11c48fa3827406c9e6304f88332e059 a8f41d79dc7a490c91013616f66c0134 - - -] [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] Instance failed to spawn<br class="">2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] Traceback (most recent call last):<br class="">2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2155, in _build_resources<br class="">2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] yield resources<br class="">2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance<br class="">2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] block_device_info=block_device_info)<br class="">2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2515, in spawn<br class="">2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] block_device_info=block_device_info)<br class="">2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4612, in _create_domain_and_network<br class="">2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] raise exception.VirtualInterfaceCreateException()<br class="">2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] VirtualInterfaceCreateException: Virtual Interface creation failed<br class="">2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] <br class="">2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [req-b5ec5b9d-2299-411f-b259-7a48b2b47d26 e11c48fa3827406c9e6304f88332e059 a8f41d79dc7a490c91013616f66c0134 - - -] [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] Failed to allocate network(s)<br class="">2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] Traceback (most recent call last):<br class="">2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance<br class="">2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] block_device_info=block_device_info)<br class="">2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2515, in spawn<br class="">2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] block_device_info=block_device_info)<br class="">2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4612, in _create_domain_and_network<br class="">2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] raise exception.VirtualInterfaceCreateException()<br class="">2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] VirtualInterfaceCreateException: Virtual Interface creation failed<br class="">2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] <br class="">2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [req-b5ec5b9d-2299-411f-b259-7a48b2b47d26 e11c48fa3827406c9e6304f88332e059 a8f41d79dc7a490c91013616f66c0134 - - -] [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] Build of instance 93bf7968-09b2-4890-9c3a-0d909ce57017 aborted: Failed to allocate the network(s), not rescheduling.<br class="">2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] Traceback (most recent call last):<br class="">2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1905, in _do_build_and_run_instance<br class="">2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] filter_properties)<br class="">2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2044, in _build_and_run_instance<br class="">2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] reason=msg)<br class="">2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] BuildAbortException: Build of instance 93bf7968-09b2-4890-9c3a-0d909ce57017 aborted: Failed to allocate the network(s), not rescheduling.<br class="">2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance: 93bf7968-09b2-4890-9c3a-0d909ce57017] <br class="">2016-04-20 18:42:55.909 860 WARNING nova.virt.libvirt.driver [req-9a205dde-ed7f-43aa-96d4-f6776347359e e11c48fa3827406c9e6304f88332e059 a8f41d79dc7a490c91013616f66c0134 - - -] [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c] Timeout waiting for vif plugging callback for instance ae94da8c-3fe6-44d6-93e5-b84a606c725c<br class="">2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [req-9a205dde-ed7f-43aa-96d4-f6776347359e e11c48fa3827406c9e6304f88332e059 a8f41d79dc7a490c91013616f66c0134 - - -] [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c] Instance failed to spawn<br class="">2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c] Traceback (most recent call last):<br class="">2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2155, in _build_resources<br class="">2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c] yield resources<br class="">2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c] File "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in _build_and_run_instance<br class="">2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c] block_device_info=block_device_info)<br class="">2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2515, in spawn<br class="">2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c] block_device_info=block_device_info)<br class="">2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c] File "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4612, in _create_domain_and_network<br class="">2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c] raise exception.VirtualInterfaceCreateException()<br class="">2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c] VirtualInterfaceCreateException: Virtual Interface creation failed<br class="">2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance: ae94da8c-3fe6-44d6-93e5-b84a606c725c]<br class=""><br class=""><br class=""></div><div class="">On the controller node, in /var/log/neutron/server.log:<br class=""><br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'tag': u'4c0a7262-ac40-4556-bb11-5b20afceec73', 'name': 'network-vif-deleted', 'server_uuid': u'93bf7968-09b2-4890-9c3a-0d909ce57017'}]<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova Traceback (most recent call last):<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/neutron/notifiers/nova.py", line 247, in send_events<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova batched_events)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/v2/contrib/server_external_events.py", line 39, in create<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return_raw=True)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 169, in _create<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova _resp, body = self.api.client.post(url, body=body)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in post<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return self.request(url, 'POST', **kwargs)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 91, in request<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova **kwargs)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return self.session.request(url, method, **kwargs)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return func(*args, **kwargs)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 304, in request<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova auth_headers = self.get_auth_headers(auth)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 617, in get_auth_headers<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return auth.get_headers(self, **kwargs)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/base.py", line 142, in get_headers<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova token = self.get_token(session)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return self.get_access(session).auth_token<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova self.auth_ref = self.get_auth_ref(session)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py", line 186, in get_auth_ref<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return self._plugin.get_auth_ref(session, **kwargs)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova authenticated=False, log=False, **rkwargs)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return self.request(url, 'POST', **kwargs)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return func(*args, **kwargs)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova raise exceptions.from_response(resp, method, url)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova BadRequest: Expecting to find id or name in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-0a9e36df-420c-46f2-a523-c168f0c295c6)<br class="">2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova <br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'status': 'completed', 'tag': u'67291fab-d2d6-42ec-ae4e-6e5ed60fa424', 'name': 'network-vif-plugged', 'server_uuid': u'ae94da8c-3fe6-44d6-93e5-b84a606c725c'}]<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova Traceback (most recent call last):<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/neutron/notifiers/nova.py", line 247, in send_events<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova batched_events)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/v2/contrib/server_external_events.py", line 39, in create<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return_raw=True)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 169, in _create<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova _resp, body = self.api.client.post(url, body=body)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in post<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return self.request(url, 'POST', **kwargs)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 91, in request<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova **kwargs)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return self.session.request(url, method, **kwargs)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return func(*args, **kwargs)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 304, in request<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova auth_headers = self.get_auth_headers(auth)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 617, in get_auth_headers<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return auth.get_headers(self, **kwargs)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/base.py", line 142, in get_headers<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova token = self.get_token(session)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return self.get_access(session).auth_token<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova self.auth_ref = self.get_auth_ref(session)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py", line 186, in get_auth_ref<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return self._plugin.get_auth_ref(session, **kwargs)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova authenticated=False, log=False, **rkwargs)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return self.request(url, 'POST', **kwargs)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return func(*args, **kwargs)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova raise exceptions.from_response(resp, method, url)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova BadRequest: Expecting to find id or name in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-7e1674f8-70f3-4c45-b5e1-3c6739bc745f)<br class="">2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova <br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova [-] Failed to notify nova on events: [{'tag': u'67291fab-d2d6-42ec-ae4e-6e5ed60fa424', 'name': 'network-vif-deleted', 'server_uuid': u'ae94da8c-3fe6-44d6-93e5-b84a606c725c'}]<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova Traceback (most recent call last):<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/neutron/notifiers/nova.py", line 247, in send_events<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova batched_events)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/v2/contrib/server_external_events.py", line 39, in create<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return_raw=True)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/base.py", line 169, in _create<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova _resp, body = self.api.client.post(url, body=body)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in post<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return self.request(url, 'POST', **kwargs)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/novaclient/client.py", line 91, in request<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova **kwargs)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return self.session.request(url, method, **kwargs)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return func(*args, **kwargs)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 304, in request<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova auth_headers = self.get_auth_headers(auth)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 617, in get_auth_headers<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return auth.get_headers(self, **kwargs)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/base.py", line 142, in get_headers<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova token = self.get_token(session)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 200, in get_token<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return self.get_access(session).auth_token<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova self.auth_ref = self.get_auth_ref(session)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py", line 186, in get_auth_ref<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return self._plugin.get_auth_ref(session, **kwargs)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova authenticated=False, log=False, **rkwargs)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return self.request(url, 'POST', **kwargs)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return func(*args, **kwargs)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova raise exceptions.from_response(resp, method, url)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova BadRequest: Expecting to find id or name in user - the server could not comply with the request since it is either malformed or otherwise incorrect. The client is assumed to be in error. (HTTP 400) (Request-ID: req-022851be-63e5-4f22-851d-60536516b686)<br class="">2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova<br class=""><br class=""><br class=""></div>It's not obvious to me how to fix this problem. Can someone please provide some guidance? What else should I look at? What service is nova calling when it times out? What is the "BadRequest" about? (where can I find the actual request? and what should it look like?)<br class=""><br class=""></div>Any help would be appreciated...<br class=""></div>Bond<br class=""></div></div></div></div></div><span class="">
_______________________________________________<br class="">Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br class="">Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank" class="">openstack@lists.openstack.org</a><br class="">Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br class=""><br class=""><br class=""><br class=""></span></div></blockquote></div><br class=""></div></div></blockquote></div><br class=""></div></div></div></div></div>
</div></blockquote></div><br class=""></div></div></blockquote></div></div></div><br class=""></div></div>
</blockquote></div><br class=""></div></div></div>
!DSPAM:1,57213fed154331981920320!
</div></blockquote></div><br class=""></div></blockquote></div><br class=""></div></div></div></div></div>
</blockquote></div><br class=""></div>
!DSPAM:1,572144f4165284957587286!
</div></blockquote></div><br class=""></div></body></html>