[Openstack] (HTTP 500) Timeout when booting instance with neutron network.

Karan digitalkaran at gmail.com
Mon Mar 23 08:43:57 UTC 2015


Thanks Sushma for looking into it. I figured out the cause of the
problem. It was due to misconfiguration in nova.conf of network node.
The official installation documentation is misleading when configuring
networking.
>From logs and by tracing the code, I figured out nova was using
nova-network api rather than neutron api.
It was because the documentation mentions to update nova.conf of
compute node but missed to mention that nova.conf of controller node
should also be updated to use neutron api. Added this in nova.conf at
controller node and now things work correctly.

[DEFAULT]

network_api_class = nova.network.neutronv2.api.API
security_group_api = neutron
linuxnet_interface_driver = nova.network.linux_net.LinuxOVSInterfaceDriver
firewall_driver = nova.virt.firewall.NoopFirewallDriver

[neutron]

url = http://controller:9696
auth_strategy = keystone
admin_auth_url = http://controller:35357/v2.0
admin_tenant_name = service
admin_username = neutron
admin_password = NEUTRON_PASS

On Mon, Mar 23, 2015 at 12:18 PM, Sushma Korati
<sushma.korati at atmecs.com> wrote:
> Hi Karan,
>
>
> Please make sure you provide private network details as argument to --net-id
> option.
>
> Also when you network, is the VM still getting IP? if so, then from which
> network is the VM getting IP address??
> Please paste the console output of "neutron net-list" and "nova boot"
> commands, it will be more helpful for debugging.
>
>  Regards,
> Sushma.
>
> On Mon, Mar 23, 2015 at 12:35 AM, digitalkaran <digitalkaran at gmail.com>
> wrote:
>>
>> Hi
>>
>> I have setup a simple 2 node setup of juno (Controller/Netwrok +
>> Compute) by following openstack installation guide.
>> However, when I try to boot an instance from cli I get following error.
>> ERROR (ClientException): The server has either erred or is incapable
>> of performing the requested operation. (HTTP 500) (Request-ID:
>> req-90390fff-fb7f-4b40-8d3a-1e70d6f22451)
>>
>> This error happens only when I provide --nic net-id to nova boot but
>> when I try to to boot without network then the instance boots
>> successfully.
>>
>> nova --debug  boot --poll  --flavor m1.tiny --image
>> cirros-0.3.3-x86_64  --security-group default demo-instance1 --nic
>> net-id=919828d6-0c40-4167-8140-bf2ae60f616f
>>
>>
>> I think the problem is with my network configuration that's why trying
>> to create instance without network  works fine.
>> By following  the official installation guide, I have created one
>> physical provider network and am trying to boot using it.
>>
>> From debug logs, nova always get stuck at this create server api call:
>>
>> REQ: curl -i
>> 'http://controller:8774/v2/f66287046354409ba7184b7aaf3f0e22/servers'
>> -X POST -H "Accept: application/json" -H "Content-Type:
>> application/json" -H "User-Agent: python-novaclient" -H
>> "X-Auth-Project-Id: admin" -H "X-Auth-Token:
>> {SHA1}0eb763e42c9dc6eb9cd5cae57c085155f4a05071" -d '{"server":
>> {"name": "demo-instance1", "imageRef":
>> "d90449ee-28ec-4618-b814-0bc9b1925218", "flavorRef": "1", "max_count":
>> 1, "min_count": 1, "networks": [{"uuid":
>> "919828d6-0c40-4167-8140-bf2ae60f616f"}], "security_groups": [{"name":
>> "default"}]}}'
>> INFO (connectionpool:259) Starting new HTTP connection (1): controller
>> DEBUG (connectionpool:390) Setting read timeout to 600.0
>>
>> And here is the traceback stack:
>>
>> Traceback (most recent call last):
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py",
>> line 124, in __call__
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     return
>> req.get_response(self.application)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/webob/request.py", line 1320, in
>> send
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     application,
>> catch_exc_info=False)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/webob/request.py", line 1284, in
>> call_application
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     app_iter =
>> application(self.environ, start_response)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     return
>> resp(environ, start_response)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/keystonemiddleware/auth_token.py",
>> line 661, in __call__
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     return
>> self._app(env, start_response)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     return
>> resp(environ, start_response)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     return
>> resp(environ, start_response)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in
>> __call__
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     response =
>> self.app(environ, start_response)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     return
>> resp(environ, start_response)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     resp =
>> self.call_func(req, *args, **self.kwargs)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in
>> call_func
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     return
>> self.func(req, *args, **kwargs)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line
>> 910, in __call__
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack
>> content_type, body, accept)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line
>> 975, in _process_stack
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack
>> action_result = self.dispatch(meth, request, action_args)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line
>> 1059, in dispatch
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     return
>> method(req=request, **action_args)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/api/openstack/compute/servers.py",
>> line 958, in create
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack
>> check_server_group_quota=check_server_group_quota)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/hooks.py", line 131, in inner
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     rv =
>> f(*args, **kwargs)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1457, in
>> create
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack
>> check_server_group_quota=check_server_group_quota)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 1079, in
>> _create_instance
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     max_count)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 770, in
>> _validate_and_build_base_options
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack
>> requested_networks, max_count)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/compute/api.py", line 479, in
>> _check_requested_networks
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     max_count)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 48, in
>> wrapped
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     return
>> func(self, context, *args, **kwargs)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 404, in
>> validate_networks
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack
>> requested_networks)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 245,
>> in validate_networks
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     return
>> self.client.call(ctxt, 'validate_networks', networks=networks)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line
>> 389, in call
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     return
>> self.prepare().call(ctxt, method, **kwargs)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/oslo/messaging/rpc/client.py", line
>> 152, in call
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack
>> retry=self.retry)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/oslo/messaging/transport.py", line
>> 90, in _send
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack
>> timeout=timeout, retry=retry)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py",
>> line 408, in send
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     retry=retry)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py",
>> line 397, in _send
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     result =
>> self._waiter.wait(msg_id, timeout)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py",
>> line 285, in wait
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     reply,
>> ending = self._poll_connection(msg_id, timeout)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack   File
>> "/usr/lib/python2.7/dist-packages/oslo/messaging/_drivers/amqpdriver.py",
>> line 235, in _poll_connection
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack     % msg_id)
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack
>> MessagingTimeout: Timed out waiting for a reply to message ID
>> 9d01a116bdc34545951bf2bc44b4e5f3
>> 2015-03-22 23:52:26.145 5244 TRACE nova.api.openstack
>> 2015-03-22 23:52:26.220 5244 INFO nova.api.openstack
>> [req-90390fff-fb7f-4b40-8d3a-1e70d6f22451 None]
>> http://controller:8774/v2/f66287046354409ba7184b7aaf3f0e22/servers
>> returned with HTTP 500
>>
>>
>>
>>
>> validate_networks seems to be a problem. I am not able to figure out
>> what I am doing wrong: either something is wrong with configuration
>> files or with neutron network. Need help to isolate the problem.
>>
>>
>> Thanks
>> Karan
>>
>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
>
>
> --
>
> Regards,
> Sushma
>
> The information in this message may be proprietary and/or confidential, and
> protected from disclosure. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this communication in error, please notify ATMECS and delete it from your
> computer.




More information about the Openstack mailing list