Open Stack

Am Mittwoch, 30. Oktober 2013, 19:02:33 schrieb Dheerendra:
> also look at the logs /var/log/nova/nova-network.log on compute node. It
> should tell whether it is able to connect to rabbit on controller.

I got now! The problem was a missing database entry on the compute node. Sorry 
guys for this stupid error, though the docs don't mention creating the entry.

Thanks for your help!
Florian

> 
> -Dheerendra
> 
> 
> On Wed, Oct 30, 2013 at 6:41 PM, Dheerendra <
> 
> dheerendra.madhusudhana at gmail.com> wrote:
> > Hi Florian
> > 
> > I observed this kind of issue when there is race condition.
> > 
> > 1. RabbitMQ is NOT running on controller-node
> > 2. You issue nova-network from compute-node. Now this step hangs.
> > 3. Just see what is happening in nova-api.log on controller(tail -f
> > /var/log/nova/nova-api.log)
> > 4. Now you start the RabbitMQ on the controller node.
> > 5. nova-network command compute returns the error. Also you will see
> > 'timeout' error in nova-api.log.
> > 
> > From your log I suspect this because initially RabitMQ was not up. Looks
> > like you started it afterwards.
> > Did you try to run the same 'nova-network' command for the second time ?
> > i.e after restarting the rabbitMQ ?
> > 
> > -Dheerendra
> > 
> > On Wed, Oct 30, 2013 at 3:47 PM, Florian Lindner 
<mailinglists at xgm.de>wrote:
> >> Am Dienstag, 29. Oktober 2013, 14:32:00 schrieb Razique Mahroua:
> >> > You are using Neutron?
> >> > Looks like the Neutron server is unable to perform the request actually
> >> 
> >> Actually no, I don't think so, I think I use the nova-network.
> >> 
> >> The documentation
> >> http://docs.openstack.org/havana/install-guide/install/apt/content/ says:
> >> 
> >> Configuring Networking can be one of the most bewildering experiences you
> >> will
> >> encounter when working with OpenStack. To assist in this we have chosen
> >> the
> >> simplest production-ready configuration for this guide: the legacy
> >> networking
> >> in OpenStack Compute, with a flat network, that takes care of DHCP.
> >> 
> >> This setup uses "multi-host" functionality: the networking is configured
> >> to be
> >> highly available by splitting networking functionality across multiple
> >> hosts.
> >> As a result, there is no single network controller that acts as a single
> >> point
> >> of failure. Because each compute node is configured for networking in
> >> this
> >> setup, no additional networking configuration is required on the
> >> controller.
> >> 
> >> (Chapter 5: Enabling Networking)
> >> 
> >> and I have installed the nova-network package.
> >> 
> >> I've downloaded my entire /etc to http://xgm.de/upload/etc/ if you or
> >> anyone
> >> else wants to have a look! Yes, these "123" are the actual passwords,
> >> it's
> >> just from a VM on my local computer.  ;-)
> >> 
> >> Thanks a lot!
> >> 
> >> Florian
> >> 
> >> > On October 29, 2013 at 12:00:18, Florian Lindner (mailinglists at xgm.de)
> >> > wrote:
> >> > 
> >> > Am Montag, 28. Oktober 2013, 13:44:00 schrieb Craig E. Ward:
> >> > > On the RabbitMQ server, are vhost and user id setup correctly?
> >> > 
> >> > Ok, part of the problem seemed to be that the documentation assumed I
> >> 
> >> set
> >> 
> >> > the password guest for the rabbit guest account. Did that now and the
> >> > problem changed:
> >> > 
> >> > root at controller:~# nova network-create vmnet --fixed-range-v4=
> >> 
> >> 10.0.0.0/24 --
> >> 
> >> > bridge-interface=br100 --multi-host=T
> >> > ERROR: The server has either erred or is incapable of performing the
> >> > requested operation. (HTTP 500) (Request-ID:
> >> > req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3)
> >> > 
> >> > nova-api.log shows:
> >> > 
> >> > http://pastebin.com/raw.php?i=13U8a4Na
> >> > 
> >> > 
> >> > 2013-10-29 19:14:18.587 1272 INFO nova.osapi_compute.wsgi.server [-]
> >> 
> >> (1272)
> >> 
> >> > accepted ('192.168.122.10', 53116)
> >> > 2013-10-29 19:14:19.207 1272 INFO nova.openstack.common.rpc.common
> >> > [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3
> >> 
> >> a1dd41c156c64810b41d46d4062a2b17
> >> 
> >> > 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on
> >> > controller:5672 2013-10-29 19:14:19.218 1272 INFO
> >> > nova.openstack.common.rpc.common
> >> 
> >> [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3
> >> 
> >> > a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f]
> >> > Connected to AMQP server on controller:5672 2013-10-29 19:15:19.238
> >> > 1272
> >> > ERROR nova.api.openstack [req-900a80bd- dfc1-49fc-9599-239fd5b9b7e3
> >> > a1dd41c156c64810b41d46d4062a2b17
> >> > 68d6cb4e2ceb4700931b6489ad25fb6f] Caught error: Timeout while waiting
> >> 
> >> on RPC
> >> 
> >> > response - topic: "network", RPC method: "create_networks" info:
> >> > "<unknown>" 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack
> >> 
> >> Traceback
> >> 
> >> > (most recent call last):
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py", line
> >> 
> >> 119,
> >> 
> >> > in __call__
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> >> > req.get_response(self.application)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in send
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack application,
> >> > catch_exc_info=False)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in
> >> > call_application
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack app_iter =
> >> > application(self.environ, start_response)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> >> 
> >> resp(environ,
> >> 
> >> > start_response)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> 
> >> "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py
> >> ",
> >> 
> >> > line 539, in __call__
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> >> 
> >> self.app(env,
> >> 
> >> > start_response)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> >> 
> >> resp(environ,
> >> 
> >> > start_response)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> >> 
> >> resp(environ,
> >> 
> >> > start_response)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131, in
> >> > __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack response
> >> 
> >> =
> >> 
> >> > self.app(environ, start_response)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in __call__
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> >> 
> >> resp(environ,
> >> 
> >> > start_response)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack resp =
> >> > self.call_func(req, *args, **self.kwargs)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> >> > self.func(req, *args, **kwargs)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line
> >> 
> >> 917, in
> >> 
> >> > __call__
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack content_type,
> >> 
> >> body,
> >> 
> >> > accept)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line
> >> 
> >> 976, in
> >> 
> >> > _process_stack
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack action_result =
> >> > self.dispatch(meth, request, action_args)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line
> >> 
> >> 1057, in
> >> 
> >> > dispatch
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> >> > method(req=request, **action_args)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-
> >> > packages/nova/api/openstack/compute/contrib/os_networks.py", line 126,
> >> 
> >> in
> >> 
> >> > create
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack network =
> >> > self.network_api.create(context, **params)[0]
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 93, in
> >> 
> >> wrapped
> >> 
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return func(self,
> >> > context, *args, **kwargs)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 142, in
> >> 
> >> create
> >> 
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> >> > self.network_rpcapi.create_networks(context, **kwargs)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line 102, in
> >> > create_networks
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> >> > self.client.call(ctxt, 'create_networks', **kwargs)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 85, in call
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> >> > self._invoke(self.proxy.call, ctxt, method, **kwargs)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 63, in
> >> 
> >> _invoke
> >> 
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> >> > cast_or_call(ctxt, msg, **self.kwargs)
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> >> > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py",
> >> 
> >> line
> >> 
> >> > 130, in call
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack exc.info,
> >> > real_topic, msg.get('method'))
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack Timeout: Timeout
> >> 
> >> while
> >> 
> >> > waiting on RPC response - topic: "network", RPC method:
> >> "create_networks"
> >> 
> >> > info: "<unknown>"
> >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack
> >> > 2013-10-29 19:15:19.249 1272 INFO nova.api.openstack [req-900a80bd-
> >> > dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17
> >> > 68d6cb4e2ceb4700931b6489ad25fb6f]
> >> > http://controller:8774/v2/68d6cb4e2ceb4700931b6489ad25fb6f/os-networks
> >> > returned with HTTP 500
> >> > 2013-10-29 19:15:19.255 1272 INFO nova.osapi_compute.wsgi.server
> >> > [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3
> >> 
> >> a1dd41c156c64810b41d46d4062a2b17
> >> 
> >> > 68d6cb4e2ceb4700931b6489ad25fb6f] 192.168.122.10 "POST
> >> > /v2/68d6cb4e2ceb4700931b6489ad25fb6f/os-networks HTTP/1.1" status: 500
> >> 
> >> len:
> >> > 335 time: 60.6673729
> >> > 
> >> > Sorry, I got not idea...
> >> > 
> >> > Thanks!
> >> > Florian
> >> > 
> >> > > % rabbitmqctl list_users
> >> > > 
> >> > > % rabbitmqctl list_vhosts
> >> > > 
> >> > > The user and vhost should match what is in the OpenStack
> >> 
> >> configurations.
> >> 
> >> > > On 10/28/13 1:14 PM, Florian Lindner wrote:
> >> > > > Am Montag, 28. Oktober 2013, 12:04:02 schrieb Razique Mahroua:
> >> > > >> hey
> >> > > >> is «controller» a hostame all ur servers can resolve?
> >> > > > 
> >> > > > Yes, set in /etc/hosts.
> >> > > > 
> >> > > > Regards,
> >> > > > Florian
> >> > > > 
> >> > > >> - Razique
> >> > > >> 
> >> > > >> On Oct 28, 2013, at 11:40, Florian Lindner <mailinglists at xgm.de>
> >> 
> >> wrote:
> >> > > >>> Hello,
> >> > > >>> 
> >> > > >>> I've installed OpenStack on a Ubuntu 12.04 VM according to the
> >> 
> >> install
> >> 
> >> > > >>> guide. It worked fine until I created the compute node on a
> >> 
> >> second VM
> >> 
> >> > > >>> and
> >> 
> >> > > >>> tried to set up nova networking:
> >> http://docs.openstack.org/havana/install-guide/install/apt/content/nov
> >> 
> >> > > >>> a-> >>> ne twork.html
> >> > > >>> 
> >> > > >>> # nova network-create vmnet --fixed-range-v4=10.0.0.0/24--bridge-
> >> > > >>> interface=br100 --multi-host=T
> >> > > >>> 
> >> > > >>> just does nothing until canceled. I tried on the controller and
> >> > > >>> compute1
> >> > > >>> node.
> >> > > >>> 
> >> > > >>> nova-api.log shows:
> >> > > >>> 
> >> > > >>> ERROR nova.openstack.common.rpc.common AMQP server on
> >> 
> >> controller:5672
> >> 
> >> > > >>> is
> >> > > >>> unreachable: Socket closed. Trying again in 1 seconds.
> >> > > >>> INFO nova.openstack.common.rpc.common Reconnecting to AMQP server
> >> 
> >> on
> >> 
> >> > > >>> controller:5672
> >> > > >>> 
> >> > > >>> controller:~# netstat -lpn | grep beam
> >> > > >>> tcp 0 0 0.0.0.0:36595 0.0.0.0:*
> >> > > >>> LISTEN
> >> > > >>> 1045/beam
> >> > > >>> tcp6 0 0 :::5672 :::*
> >> > > >>> LISTEN
> >> > > >>> 1045/beam
> >> > > >>> 
> >> > > >>> It listens on this port, but IPv6. IMO on linux listing on an
> >> > > >>> IPv6
> >> > > >>> port
> >> > > >>> also means listening on the same v4 port.
> >> > > >>> 
> >> > > >>> What's wrong with the installation?
> >> 
> >> _______________________________________________
> >> Mailing list:
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >> Post to     : openstack at lists.openstack.org
> >> Unsubscribe :
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack