[Openstack] Help with Openstack Liberty: Failed to allocate the network
The O.G.
theoriginalguru at gmail.com
Wed Apr 27 23:01:35 UTC 2016
Remo & other OpenStack:
I've been looking into this more today and noticed this WARNING message
from the nova-compute.log (compute node):
2016-04-27 15:39:39.191 856 WARNING oslo_config.cfg [-] Option "username"
from group "neutron" is deprecated. Use option "user-name" from group
"neutron".
This message is written around the same time as this message in neutron
server.log (controller node):
2016-04-27 15:41:57.398 2262 ERROR neutron.notifiers.nova BadRequest:
Expecting to find id or name in user - the server could not comply with the
request since it is either malformed or otherwise incorrect. The client is
assumed to be in error. (HTTP 400) (Request-ID:
req-854b955c-2732-464b-b8b5-d087bf4ef9f8)
Also, noticed similar message in keystone.log (controller node):
2016-04-27 15:46:57.167 4593 WARNING keystone.common.wsgi
[req-465cfd8f-194e-47be-a83f-fe8ae90b8766 - - - - -] Expecting to find id
or name in user - the server could not comply with the request since it is
either malformed or otherwise incorrect. The client is assumed to be in
error.
Can anyone please help me narrow down where I might be able to find the
cause of these error messages?
Thanks.
On Wed, Apr 27, 2016 at 3:56 PM, The O.G. <theoriginalguru at gmail.com> wrote:
> Yes, this is on both controller and compute node.
>
>
> On Wed, Apr 27, 2016 at 3:41 PM, Remo Mattei <remo at italy1.com> wrote:
>
>> is this on the controller and computes?
>>
>> On Apr 27, 2016, at 15:40, The O.G. <theoriginalguru at gmail.com> wrote:
>>
>> To be more specific, SELinux is enabled, but in permissive mode:
>>
>> # sestatus
>> SELinux status: enabled
>> SELinuxfs mount: /sys/fs/selinux
>> SELinux root directory: /etc/selinux
>> Loaded policy name: targeted
>> Current mode: permissive
>> Mode from config file: permissive
>> Policy MLS status: enabled
>> Policy deny_unknown status: allowed
>> Max kernel policy version: 28
>>
>>
>> On Wed, Apr 27, 2016 at 3:36 PM, The O.G. <theoriginalguru at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Wed, Apr 27, 2016 at 3:34 PM, Remo Mattei <remo at italy1.com> wrote:
>>>
>>>> are you running SELinux?
>>>> sestatus on all of them and let me know.
>>>>
>>>> yes, SELinux enabled on all systems. are you suggesting I try with
>>> SELinux disabled? (I can do that...)
>>>
>>> thanks...
>>>
>>> Remo
>>>>
>>>> On Apr 27, 2016, at 15:23, The O.G. <theoriginalguru at gmail.com> wrote:
>>>>
>>>> Hi Remo,
>>>>
>>>> On Thu, Apr 21, 2016 at 7:00 AM, Remo Mattei <remo at italy1.com> wrote:
>>>>
>>>>> has this been a new installation or an upgrade?
>>>>>
>>>>
>>>> This is a brand new installation of Openstack/Liberty on CentOS7
>>>>
>>>>
>>>>> Not sure what you mean with option 2 since the install guide does not
>>>>> say anything about options.
>>>>>
>>>>
>>>> If you see here:
>>>>
>>>>
>>>> http://docs.openstack.org/liberty/install-guide-rdo/neutron-controller-install.html
>>>>
>>>> Under "Configure networking options", there's an option 1 (Provider
>>>> network) and option 2 (self-service network). All I'm saying is that we
>>>> have been using "option 2" while following the guide.
>>>>
>>>> did you check if rabbit is up?
>>>>>
>>>>
>>>> Yes. We repeated all the "verify operations" steps in the guide to be
>>>> sure that the services are working. All verification steps passed. I assume
>>>> this means rabbit is up or it would not have passed. The rabbit process is
>>>> running also. Is there some other step you recommend to verify the proper
>>>> operation of rabbitmq?
>>>>
>>>> did you check and see if the services are all good
>>>>>
>>>>
>>>> Well, something is *not* good, hence we have problems. But, if you mean
>>>> by "systemctl status <service_name>", then yes, all services appear to be
>>>> good. Is there certain commands you would recommend other than systemctl
>>>> status to check on services?
>>>>
>>>>
>>>>> did you check and see if the agents are happy?
>>>>>
>>>>
>>>> I don't know what "happy" means, but this is the output of neutron
>>>> agent-list:
>>>>
>>>> # neutron agent-list
>>>>
>>>> +--------------------------------------+--------------------+-------------------------------+-------+----------------+---------------------------+
>>>> | id | agent_type |
>>>> host | alive | admin_state_up |
>>>> binary |
>>>>
>>>> +--------------------------------------+--------------------+-------------------------------+-------+----------------+---------------------------+
>>>> | 321c6188-0a8b-435e-98c7-42fcd2b9f9de | Linux bridge agent |
>>>> controller | :-) | True | neutron-linuxbridge-agent |
>>>> | 3cc20152-3db7-4a63-8b9e-530cbf272ab3 | Linux bridge agent | compute
>>>> | :-) | True | neutron-linuxbridge-agent |
>>>> | 60351fc0-873d-4fa3-b295-2146174e567c | Metadata agent |
>>>> controller | :-) | True | neutron-metadata-agent |
>>>> | a2dcef04-8ccf-4034-a7f9-4558cc8798f9 | L3 agent |
>>>> controller | :-) | True | neutron-l3-agent |
>>>> | d8aed254-edd0-4f6a-a264-4953a1a2c7e8 | DHCP agent |
>>>> controller | :-) | True | neutron-dhcp-agent |
>>>>
>>>> +--------------------------------------+--------------------+-------------------------------+-------+----------------+---------------------------+
>>>>
>>>> If the smily face :-) means happy, then yes I think they are happy. Is
>>>> there something else I should check?
>>>>
>>>>
>>>>> just a few things to check let us know!
>>>>>
>>>>>
>>>> Thanks for your reply. Do you have any hints or insight on the
>>>> "BadRequest" message in the logs I posted ?
>>>>
>>>> Bond
>>>>
>>>> Remo
>>>>>
>>>>> On Apr 20, 2016, at 19:05, The O.G. <theoriginalguru at gmail.com> wrote:
>>>>>
>>>>> We are following:
>>>>>
>>>>> http://docs.openstack.org/liberty/install-guide-rdo/index.html
>>>>>
>>>>> Using CentOS7. We've chosen the option 2 for networking, where we have
>>>>> both a private and public network. We've pretty much following the guide
>>>>> step by step up to and finishing "Add the dashboard" step. All the
>>>>> verification steps passed.
>>>>>
>>>>> Now, we are trying to launch an instance on the public and private
>>>>> networks as outlined here:
>>>>>
>>>>>
>>>>> http://docs.openstack.org/liberty/install-guide-rdo/launch-instance-public.html
>>>>>
>>>>>
>>>>> http://docs.openstack.org/liberty/install-guide-rdo/launch-instance-private.html
>>>>>
>>>>> In both attempts to launch an instance, the instance gets aborted and
>>>>> shows status = Error. With this message:
>>>>>
>>>>> Build of instance 93bf7968-09b2-4890-9c3a-0d909ce57017 aborted: Failed
>>>>> to allocate the network(s), not rescheduling.
>>>>>
>>>>> On the compute node, in /var/log/nova/nova-compute.log, we get these
>>>>> messages (I'm filtering on ERROR|WARNING only):
>>>>>
>>>>> 2016-04-20 18:35:31.484 860 WARNING nova.virt.libvirt.driver
>>>>> [req-b5ec5b9d-2299-411f-b259-7a48b2b47d26 e11c48fa3827406c9e6304f88332e059
>>>>> a8f41d79dc7a490c91013616f66c0134 - - -] [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] Timeout waiting for vif plugging
>>>>> callback for instance 93bf7968-09b2-4890-9c3a-0d909ce57017
>>>>> 2016-04-20 18:35:32.007 860 ERROR nova.compute.manager
>>>>> [req-b5ec5b9d-2299-411f-b259-7a48b2b47d26 e11c48fa3827406c9e6304f88332e059
>>>>> a8f41d79dc7a490c91013616f66c0134 - - -] [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] Instance failed to spawn
>>>>> 2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] Traceback (most recent call last):
>>>>> 2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] File
>>>>> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2155, in
>>>>> _build_resources
>>>>> 2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] yield resources
>>>>> 2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] File
>>>>> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in
>>>>> _build_and_run_instance
>>>>> 2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017]
>>>>> block_device_info=block_device_info)
>>>>> 2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] File
>>>>> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2515,
>>>>> in spawn
>>>>> 2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017]
>>>>> block_device_info=block_device_info)
>>>>> 2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] File
>>>>> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4612,
>>>>> in _create_domain_and_network
>>>>> 2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] raise
>>>>> exception.VirtualInterfaceCreateException()
>>>>> 2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] VirtualInterfaceCreateException:
>>>>> Virtual Interface creation failed
>>>>> 2016-04-20 18:35:32.007 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017]
>>>>> 2016-04-20 18:35:32.587 860 ERROR nova.compute.manager
>>>>> [req-b5ec5b9d-2299-411f-b259-7a48b2b47d26 e11c48fa3827406c9e6304f88332e059
>>>>> a8f41d79dc7a490c91013616f66c0134 - - -] [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] Failed to allocate network(s)
>>>>> 2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] Traceback (most recent call last):
>>>>> 2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] File
>>>>> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in
>>>>> _build_and_run_instance
>>>>> 2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017]
>>>>> block_device_info=block_device_info)
>>>>> 2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] File
>>>>> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2515,
>>>>> in spawn
>>>>> 2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017]
>>>>> block_device_info=block_device_info)
>>>>> 2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] File
>>>>> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4612,
>>>>> in _create_domain_and_network
>>>>> 2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] raise
>>>>> exception.VirtualInterfaceCreateException()
>>>>> 2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] VirtualInterfaceCreateException:
>>>>> Virtual Interface creation failed
>>>>> 2016-04-20 18:35:32.587 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017]
>>>>> 2016-04-20 18:35:32.590 860 ERROR nova.compute.manager
>>>>> [req-b5ec5b9d-2299-411f-b259-7a48b2b47d26 e11c48fa3827406c9e6304f88332e059
>>>>> a8f41d79dc7a490c91013616f66c0134 - - -] [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] Build of instance
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017 aborted: Failed to allocate the
>>>>> network(s), not rescheduling.
>>>>> 2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] Traceback (most recent call last):
>>>>> 2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] File
>>>>> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 1905, in
>>>>> _do_build_and_run_instance
>>>>> 2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] filter_properties)
>>>>> 2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] File
>>>>> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2044, in
>>>>> _build_and_run_instance
>>>>> 2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] reason=msg)
>>>>> 2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017] BuildAbortException: Build of
>>>>> instance 93bf7968-09b2-4890-9c3a-0d909ce57017 aborted: Failed to allocate
>>>>> the network(s), not rescheduling.
>>>>> 2016-04-20 18:35:32.590 860 ERROR nova.compute.manager [instance:
>>>>> 93bf7968-09b2-4890-9c3a-0d909ce57017]
>>>>> 2016-04-20 18:42:55.909 860 WARNING nova.virt.libvirt.driver
>>>>> [req-9a205dde-ed7f-43aa-96d4-f6776347359e e11c48fa3827406c9e6304f88332e059
>>>>> a8f41d79dc7a490c91013616f66c0134 - - -] [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c] Timeout waiting for vif plugging
>>>>> callback for instance ae94da8c-3fe6-44d6-93e5-b84a606c725c
>>>>> 2016-04-20 18:42:56.358 860 ERROR nova.compute.manager
>>>>> [req-9a205dde-ed7f-43aa-96d4-f6776347359e e11c48fa3827406c9e6304f88332e059
>>>>> a8f41d79dc7a490c91013616f66c0134 - - -] [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c] Instance failed to spawn
>>>>> 2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c] Traceback (most recent call last):
>>>>> 2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c] File
>>>>> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2155, in
>>>>> _build_resources
>>>>> 2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c] yield resources
>>>>> 2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c] File
>>>>> "/usr/lib/python2.7/site-packages/nova/compute/manager.py", line 2009, in
>>>>> _build_and_run_instance
>>>>> 2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c]
>>>>> block_device_info=block_device_info)
>>>>> 2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c] File
>>>>> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2515,
>>>>> in spawn
>>>>> 2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c]
>>>>> block_device_info=block_device_info)
>>>>> 2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c] File
>>>>> "/usr/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 4612,
>>>>> in _create_domain_and_network
>>>>> 2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c] raise
>>>>> exception.VirtualInterfaceCreateException()
>>>>> 2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c] VirtualInterfaceCreateException:
>>>>> Virtual Interface creation failed
>>>>> 2016-04-20 18:42:56.358 860 ERROR nova.compute.manager [instance:
>>>>> ae94da8c-3fe6-44d6-93e5-b84a606c725c]
>>>>>
>>>>>
>>>>> On the controller node, in /var/log/neutron/server.log:
>>>>>
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova [-] Failed
>>>>> to notify nova on events: [{'tag': u'4c0a7262-ac40-4556-bb11-5b20afceec73',
>>>>> 'name': 'network-vif-deleted', 'server_uuid':
>>>>> u'93bf7968-09b2-4890-9c3a-0d909ce57017'}]
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova Traceback
>>>>> (most recent call last):
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/neutron/notifiers/nova.py", line 247, in
>>>>> send_events
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova
>>>>> batched_events)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/novaclient/v2/contrib/server_external_events.py",
>>>>> line 39, in create
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova
>>>>> return_raw=True)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/novaclient/base.py", line 169, in _create
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova _resp,
>>>>> body = self.api.client.post(url, body=body)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in
>>>>> post
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return
>>>>> self.request(url, 'POST', **kwargs)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/novaclient/client.py", line 91, in request
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova **kwargs)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in
>>>>> request
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova resp =
>>>>> super(LegacyJsonAdapter, self).request(*args, **kwargs)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in
>>>>> request
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return
>>>>> self.session.request(url, method, **kwargs)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in
>>>>> inner
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return
>>>>> func(*args, **kwargs)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 304, in
>>>>> request
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova
>>>>> auth_headers = self.get_auth_headers(auth)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 617, in
>>>>> get_auth_headers
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return
>>>>> auth.get_headers(self, **kwargs)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/base.py", line 142,
>>>>> in get_headers
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova token =
>>>>> self.get_token(session)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py",
>>>>> line 200, in get_token
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return
>>>>> self.get_access(session).auth_token
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py",
>>>>> line 240, in get_access
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova
>>>>> self.auth_ref = self.get_auth_ref(session)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py",
>>>>> line 186, in get_auth_ref
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return
>>>>> self._plugin.get_auth_ref(session, **kwargs)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py",
>>>>> line 190, in get_auth_ref
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova
>>>>> authenticated=False, log=False, **rkwargs)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in
>>>>> post
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return
>>>>> self.request(url, 'POST', **kwargs)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in
>>>>> inner
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova return
>>>>> func(*args, **kwargs)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in
>>>>> request
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova raise
>>>>> exceptions.from_response(resp, method, url)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova BadRequest:
>>>>> Expecting to find id or name in user - the server could not comply with the
>>>>> request since it is either malformed or otherwise incorrect. The client is
>>>>> assumed to be in error. (HTTP 400) (Request-ID:
>>>>> req-0a9e36df-420c-46f2-a523-c168f0c295c6)
>>>>> 2016-04-20 18:35:35.023 2699 ERROR neutron.notifiers.nova
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova [-] Failed
>>>>> to notify nova on events: [{'status': 'completed', 'tag':
>>>>> u'67291fab-d2d6-42ec-ae4e-6e5ed60fa424', 'name': 'network-vif-plugged',
>>>>> 'server_uuid': u'ae94da8c-3fe6-44d6-93e5-b84a606c725c'}]
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova Traceback
>>>>> (most recent call last):
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/neutron/notifiers/nova.py", line 247, in
>>>>> send_events
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova
>>>>> batched_events)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/novaclient/v2/contrib/server_external_events.py",
>>>>> line 39, in create
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova
>>>>> return_raw=True)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/novaclient/base.py", line 169, in _create
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova _resp,
>>>>> body = self.api.client.post(url, body=body)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in
>>>>> post
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return
>>>>> self.request(url, 'POST', **kwargs)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/novaclient/client.py", line 91, in request
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova **kwargs)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in
>>>>> request
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova resp =
>>>>> super(LegacyJsonAdapter, self).request(*args, **kwargs)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in
>>>>> request
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return
>>>>> self.session.request(url, method, **kwargs)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in
>>>>> inner
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return
>>>>> func(*args, **kwargs)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 304, in
>>>>> request
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova
>>>>> auth_headers = self.get_auth_headers(auth)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 617, in
>>>>> get_auth_headers
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return
>>>>> auth.get_headers(self, **kwargs)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/base.py", line 142,
>>>>> in get_headers
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova token =
>>>>> self.get_token(session)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py",
>>>>> line 200, in get_token
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return
>>>>> self.get_access(session).auth_token
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py",
>>>>> line 240, in get_access
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova
>>>>> self.auth_ref = self.get_auth_ref(session)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py",
>>>>> line 186, in get_auth_ref
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return
>>>>> self._plugin.get_auth_ref(session, **kwargs)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py",
>>>>> line 190, in get_auth_ref
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova
>>>>> authenticated=False, log=False, **rkwargs)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in
>>>>> post
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return
>>>>> self.request(url, 'POST', **kwargs)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in
>>>>> inner
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova return
>>>>> func(*args, **kwargs)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in
>>>>> request
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova raise
>>>>> exceptions.from_response(resp, method, url)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova BadRequest:
>>>>> Expecting to find id or name in user - the server could not comply with the
>>>>> request since it is either malformed or otherwise incorrect. The client is
>>>>> assumed to be in error. (HTTP 400) (Request-ID:
>>>>> req-7e1674f8-70f3-4c45-b5e1-3c6739bc745f)
>>>>> 2016-04-20 18:38:00.063 2701 ERROR neutron.notifiers.nova
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova [-] Failed
>>>>> to notify nova on events: [{'tag': u'67291fab-d2d6-42ec-ae4e-6e5ed60fa424',
>>>>> 'name': 'network-vif-deleted', 'server_uuid':
>>>>> u'ae94da8c-3fe6-44d6-93e5-b84a606c725c'}]
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova Traceback
>>>>> (most recent call last):
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/neutron/notifiers/nova.py", line 247, in
>>>>> send_events
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova
>>>>> batched_events)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/novaclient/v2/contrib/server_external_events.py",
>>>>> line 39, in create
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova
>>>>> return_raw=True)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/novaclient/base.py", line 169, in _create
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova _resp,
>>>>> body = self.api.client.post(url, body=body)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 176, in
>>>>> post
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return
>>>>> self.request(url, 'POST', **kwargs)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/novaclient/client.py", line 91, in request
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova **kwargs)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in
>>>>> request
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova resp =
>>>>> super(LegacyJsonAdapter, self).request(*args, **kwargs)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in
>>>>> request
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return
>>>>> self.session.request(url, method, **kwargs)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in
>>>>> inner
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return
>>>>> func(*args, **kwargs)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 304, in
>>>>> request
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova
>>>>> auth_headers = self.get_auth_headers(auth)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 617, in
>>>>> get_auth_headers
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return
>>>>> auth.get_headers(self, **kwargs)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/base.py", line 142,
>>>>> in get_headers
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova token =
>>>>> self.get_token(session)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py",
>>>>> line 200, in get_token
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return
>>>>> self.get_access(session).auth_token
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py",
>>>>> line 240, in get_access
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova
>>>>> self.auth_ref = self.get_auth_ref(session)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/generic/base.py",
>>>>> line 186, in get_auth_ref
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return
>>>>> self._plugin.get_auth_ref(session, **kwargs)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py",
>>>>> line 190, in get_auth_ref
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova
>>>>> authenticated=False, log=False, **rkwargs)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in
>>>>> post
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return
>>>>> self.request(url, 'POST', **kwargs)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in
>>>>> inner
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova return
>>>>> func(*args, **kwargs)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova File
>>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in
>>>>> request
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova raise
>>>>> exceptions.from_response(resp, method, url)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova BadRequest:
>>>>> Expecting to find id or name in user - the server could not comply with the
>>>>> request since it is either malformed or otherwise incorrect. The client is
>>>>> assumed to be in error. (HTTP 400) (Request-ID:
>>>>> req-022851be-63e5-4f22-851d-60536516b686)
>>>>> 2016-04-20 18:42:59.448 2699 ERROR neutron.notifiers.nova
>>>>>
>>>>>
>>>>> It's not obvious to me how to fix this problem. Can someone please
>>>>> provide some guidance? What else should I look at? What service is nova
>>>>> calling when it times out? What is the "BadRequest" about? (where can I
>>>>> find the actual request? and what should it look like?)
>>>>>
>>>>> Any help would be appreciated...
>>>>> Bond
>>>>> _______________________________________________
>>>>> Mailing list:
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>> Post to : openstack at lists.openstack.org
>>>>> Unsubscribe :
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>> !DSPAM:1,57213da0149221297419556!
>>>>
>>>>
>>>>
>>>
>> !DSPAM:1,57213fed154331981920320!
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160427/9d5d6d84/attachment.html>
More information about the Openstack
mailing list