[Openstack] Can't connect to RabbitMQ

Florian Lindner mailinglists at xgm.de
Thu Oct 31 11:17:35 UTC 2013


Am Mittwoch, 30. Oktober 2013, 22:35:34 schrieb Dheerendra:
> Good. Can you share the configuration file-nova.conf on compute ? It will
> help everybody.

Of course:

root at compute1:~# cat /etc/nova/nova.conf
[DEFAULT]
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
iscsi_helper=tgtadm
libvirt_use_virtio_for_bridges=True
connection_type=libvirt
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
verbose=True
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
volumes_path=/var/lib/nova/volumes
enabled_apis=ec2,osapi_compute,metadata
my_ip=192.168.122.11
vncserver_listen=0.0.0.0
vncserver_proxyclient_address=192.168.122.11
glance_host=controller

network_manager=nova.network.manager.FlatDHCPManager
firewall_driver=nova.virt.libvirt.firewall.IptablesFirewallDriver
network_size=254
allow_same_net_traffic=False
multi_host=True
send_arp_for_ha=True
share_dhcp_address=True
force_dhcp_release=True
flat_network_bridge=br100
flat_interface=eth0
public_interface=eth0

rpc_backend = nova.rpc.impl_kombu
rabbit_host = controller
rabbit_port = 5672

[database]
# The SQLAlchemy connection string used to connect to the database
connection = mysql://nova:123@controller/nova

Regards,
Florian

> 
> -Dheerendra
> 
> On Wed, Oct 30, 2013 at 8:25 PM, Florian Lindner <mailinglists at xgm.de>wrote:
> > Am Mittwoch, 30. Oktober 2013, 19:02:33 schrieb Dheerendra:
> > > also look at the logs /var/log/nova/nova-network.log on compute node. It
> > > should tell whether it is able to connect to rabbit on controller.
> > 
> > I got now! The problem was a missing database entry on the compute node.
> > Sorry
> > guys for this stupid error, though the docs don't mention creating the
> > entry.
> > 
> > Thanks for your help!
> > Florian
> > 
> > > -Dheerendra
> > > 
> > > 
> > > On Wed, Oct 30, 2013 at 6:41 PM, Dheerendra <
> > > 
> > > dheerendra.madhusudhana at gmail.com> wrote:
> > > > Hi Florian
> > > > 
> > > > I observed this kind of issue when there is race condition.
> > > > 
> > > > 1. RabbitMQ is NOT running on controller-node
> > > > 2. You issue nova-network from compute-node. Now this step hangs.
> > > > 3. Just see what is happening in nova-api.log on controller(tail -f
> > > > /var/log/nova/nova-api.log)
> > > > 4. Now you start the RabbitMQ on the controller node.
> > > > 5. nova-network command compute returns the error. Also you will see
> > > > 'timeout' error in nova-api.log.
> > > > 
> > > > From your log I suspect this because initially RabitMQ was not up.
> > 
> > Looks
> > 
> > > > like you started it afterwards.
> > > > Did you try to run the same 'nova-network' command for the second time
> > 
> > ?
> > 
> > > > i.e after restarting the rabbitMQ ?
> > > > 
> > > > -Dheerendra
> > > > 
> > > > On Wed, Oct 30, 2013 at 3:47 PM, Florian Lindner
> > 
> > <mailinglists at xgm.de>wrote:
> > > >> Am Dienstag, 29. Oktober 2013, 14:32:00 schrieb Razique Mahroua:
> > > >> > You are using Neutron?
> > > >> > Looks like the Neutron server is unable to perform the request
> > 
> > actually
> > 
> > > >> Actually no, I don't think so, I think I use the nova-network.
> > > >> 
> > > >> The documentation
> > > >> http://docs.openstack.org/havana/install-guide/install/apt/content/sa
> > > >> ys:
> > > >> 
> > > >> Configuring Networking can be one of the most bewildering experiences
> > 
> > you
> > 
> > > >> will
> > > >> encounter when working with OpenStack. To assist in this we have
> > 
> > chosen
> > 
> > > >> the
> > > >> simplest production-ready configuration for this guide: the legacy
> > > >> networking
> > > >> in OpenStack Compute, with a flat network, that takes care of DHCP.
> > > >> 
> > > >> This setup uses "multi-host" functionality: the networking is
> > 
> > configured
> > 
> > > >> to be
> > > >> highly available by splitting networking functionality across
> > > >> multiple
> > > >> hosts.
> > > >> As a result, there is no single network controller that acts as a
> > 
> > single
> > 
> > > >> point
> > > >> of failure. Because each compute node is configured for networking in
> > > >> this
> > > >> setup, no additional networking configuration is required on the
> > > >> controller.
> > > >> 
> > > >> (Chapter 5: Enabling Networking)
> > > >> 
> > > >> and I have installed the nova-network package.
> > > >> 
> > > >> I've downloaded my entire /etc to http://xgm.de/upload/etc/ if you or
> > > >> anyone
> > > >> else wants to have a look! Yes, these "123" are the actual passwords,
> > > >> it's
> > > >> just from a VM on my local computer.  ;-)
> > > >> 
> > > >> Thanks a lot!
> > > >> 
> > > >> Florian
> > > >> 
> > > >> > On October 29, 2013 at 12:00:18, Florian Lindner (
> > 
> > mailinglists at xgm.de)
> > 
> > > >> > wrote:
> > > >> > 
> > > >> > Am Montag, 28. Oktober 2013, 13:44:00 schrieb Craig E. Ward:
> > > >> > > On the RabbitMQ server, are vhost and user id setup correctly?
> > > >> > 
> > > >> > Ok, part of the problem seemed to be that the documentation assumed
> > 
> > I
> > 
> > > >> set
> > > >> 
> > > >> > the password guest for the rabbit guest account. Did that now and
> > 
> > the
> > 
> > > >> > problem changed:
> > > >> > 
> > > >> > root at controller:~# nova network-create vmnet --fixed-range-v4=
> > > >> 
> > > >> 10.0.0.0/24 --
> > > >> 
> > > >> > bridge-interface=br100 --multi-host=T
> > > >> > ERROR: The server has either erred or is incapable of performing
> > > >> > the
> > > >> > requested operation. (HTTP 500) (Request-ID:
> > > >> > req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3)
> > > >> > 
> > > >> > nova-api.log shows:
> > > >> > 
> > > >> > http://pastebin.com/raw.php?i=13U8a4Na
> > > >> > 
> > > >> > 
> > > >> > 2013-10-29 19:14:18.587 1272 INFO nova.osapi_compute.wsgi.server
> > > >> > [-]
> > > >> 
> > > >> (1272)
> > > >> 
> > > >> > accepted ('192.168.122.10', 53116)
> > > >> > 2013-10-29 19:14:19.207 1272 INFO nova.openstack.common.rpc.common
> > > >> > [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3
> > > >> 
> > > >> a1dd41c156c64810b41d46d4062a2b17
> > > >> 
> > > >> > 68d6cb4e2ceb4700931b6489ad25fb6f] Connected to AMQP server on
> > > >> > controller:5672 2013-10-29 19:14:19.218 1272 INFO
> > > >> > nova.openstack.common.rpc.common
> > > >> 
> > > >> [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3
> > > >> 
> > > >> > a1dd41c156c64810b41d46d4062a2b17 68d6cb4e2ceb4700931b6489ad25fb6f]
> > > >> > Connected to AMQP server on controller:5672 2013-10-29 19:15:19.238
> > > >> > 1272
> > > >> > ERROR nova.api.openstack [req-900a80bd- dfc1-49fc-9599-239fd5b9b7e3
> > > >> > a1dd41c156c64810b41d46d4062a2b17
> > > >> > 68d6cb4e2ceb4700931b6489ad25fb6f] Caught error: Timeout while
> > 
> > waiting
> > 
> > > >> on RPC
> > > >> 
> > > >> > response - topic: "network", RPC method: "create_networks" info:
> > > >> > "<unknown>" 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack
> > > >> 
> > > >> Traceback
> > > >> 
> > > >> > (most recent call last):
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/nova/api/openstack/__init__.py",
> > 
> > line
> > 
> > > >> 119,
> > > >> 
> > > >> > in __call__
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > > >> > req.get_response(self.application)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/webob/request.py", line 1296, in
> > 
> > send
> > 
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack application,
> > > >> > catch_exc_info=False)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/webob/request.py", line 1260, in
> > > >> > call_application
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack app_iter =
> > > >> > application(self.environ, start_response)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in
> > 
> > __call__
> > 
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > > >> 
> > > >> resp(environ,
> > > >> 
> > > >> > start_response)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > 
> > "/usr/lib/python2.7/dist-packages/keystoneclient/middleware/auth_token.py
> > 
> > > >> ",
> > > >> 
> > > >> > line 539, in __call__
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > > >> 
> > > >> self.app(env,
> > > >> 
> > > >> > start_response)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in
> > 
> > __call__
> > 
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > > >> 
> > > >> resp(environ,
> > > >> 
> > > >> > start_response)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in
> > 
> > __call__
> > 
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > > >> 
> > > >> resp(environ,
> > > >> 
> > > >> > start_response)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/routes/middleware.py", line 131,
> > 
> > in
> > 
> > > >> > __call__ 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack
> > 
> > response
> > 
> > > >> =
> > > >> 
> > > >> > self.app(environ, start_response)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 144, in
> > 
> > __call__
> > 
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > > >> 
> > > >> resp(environ,
> > > >> 
> > > >> > start_response)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 130, in
> > 
> > __call__
> > 
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack resp =
> > > >> > self.call_func(req, *args, **self.kwargs)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/webob/dec.py", line 195, in
> > 
> > call_func
> > 
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > > >> > self.func(req, *args, **kwargs)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line
> > > >> 
> > > >> 917, in
> > > >> 
> > > >> > __call__
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack content_type,
> > > >> 
> > > >> body,
> > > >> 
> > > >> > accept)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line
> > > >> 
> > > >> 976, in
> > > >> 
> > > >> > _process_stack
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack action_result
> > 
> > =
> > 
> > > >> > self.dispatch(meth, request, action_args)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py", line
> > > >> 
> > > >> 1057, in
> > > >> 
> > > >> > dispatch
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > > >> > method(req=request, **action_args)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-
> > > >> > packages/nova/api/openstack/compute/contrib/os_networks.py", line
> > 
> > 126,
> > 
> > > >> in
> > > >> 
> > > >> > create
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack network =
> > > >> > self.network_api.create(context, **params)[0]
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 93, in
> > > >> 
> > > >> wrapped
> > > >> 
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > 
> > func(self,
> > 
> > > >> > context, *args, **kwargs)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/nova/network/api.py", line 142,
> > > >> > in
> > > >> 
> > > >> create
> > > >> 
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > > >> > self.network_rpcapi.create_networks(context, **kwargs)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/nova/network/rpcapi.py", line
> > 
> > 102, in
> > 
> > > >> > create_networks
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > > >> > self.client.call(ctxt, 'create_networks', **kwargs)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 85, in
> > 
> > call
> > 
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > > >> > self._invoke(self.proxy.call, ctxt, method, **kwargs)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > > >> > "/usr/lib/python2.7/dist-packages/nova/rpcclient.py", line 63, in
> > > >> 
> > > >> _invoke
> > > >> 
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack return
> > > >> > cast_or_call(ctxt, msg, **self.kwargs)
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack File
> > 
> > "/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/proxy.py",
> > 
> > > >> line
> > > >> 
> > > >> > 130, in call
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack exc.info,
> > > >> > real_topic, msg.get('method'))
> > 
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack Timeout:
> > Timeout
> > 
> > > >> while
> > > >> 
> > > >> > waiting on RPC response - topic: "network", RPC method:
> > > >> "create_networks"
> > > >> 
> > > >> > info: "<unknown>"
> > > >> > 2013-10-29 19:15:19.238 1272 TRACE nova.api.openstack
> > > >> > 2013-10-29 19:15:19.249 1272 INFO nova.api.openstack [req-900a80bd-
> > > >> > dfc1-49fc-9599-239fd5b9b7e3 a1dd41c156c64810b41d46d4062a2b17
> > > >> > 68d6cb4e2ceb4700931b6489ad25fb6f]
> > 
> > http://controller:8774/v2/68d6cb4e2ceb4700931b6489ad25fb6f/os-networks
> > 
> > > >> > returned with HTTP 500
> > > >> > 2013-10-29 19:15:19.255 1272 INFO nova.osapi_compute.wsgi.server
> > > >> > [req-900a80bd-dfc1-49fc-9599-239fd5b9b7e3
> > > >> 
> > > >> a1dd41c156c64810b41d46d4062a2b17
> > > >> 
> > > >> > 68d6cb4e2ceb4700931b6489ad25fb6f] 192.168.122.10 "POST
> > 
> > > >> > /v2/68d6cb4e2ceb4700931b6489ad25fb6f/os-networks HTTP/1.1" status:
> > 500
> > 
> > > >> len:
> > > >> > 335 time: 60.6673729
> > > >> > 
> > > >> > Sorry, I got not idea...
> > > >> > 
> > > >> > Thanks!
> > > >> > Florian
> > > >> > 
> > > >> > > % rabbitmqctl list_users
> > > >> > > 
> > > >> > > % rabbitmqctl list_vhosts
> > > >> > > 
> > > >> > > The user and vhost should match what is in the OpenStack
> > > >> 
> > > >> configurations.
> > > >> 
> > > >> > > On 10/28/13 1:14 PM, Florian Lindner wrote:
> > > >> > > > Am Montag, 28. Oktober 2013, 12:04:02 schrieb Razique Mahroua:
> > > >> > > >> hey
> > > >> > > >> is «controller» a hostame all ur servers can resolve?
> > > >> > > > 
> > > >> > > > Yes, set in /etc/hosts.
> > > >> > > > 
> > > >> > > > Regards,
> > > >> > > > Florian
> > > >> > > > 
> > > >> > > >> - Razique
> > > >> > > >> 
> > > >> > > >> On Oct 28, 2013, at 11:40, Florian Lindner <
> > 
> > mailinglists at xgm.de>
> > 
> > > >> wrote:
> > > >> > > >>> Hello,
> > > >> > > >>> 
> > > >> > > >>> I've installed OpenStack on a Ubuntu 12.04 VM according to
> > > >> > > >>> the
> > > >> 
> > > >> install
> > > >> 
> > > >> > > >>> guide. It worked fine until I created the compute node on a
> > > >> 
> > > >> second VM
> > > >> 
> > > >> > > >>> and
> > 
> > > >> > > >>> tried to set up nova networking:
> > http://docs.openstack.org/havana/install-guide/install/apt/content/nov
> > 
> > > >> > > >>> a-> >>> ne twork.html
> > > >> > > >>> 
> > > >> > > >>> # nova network-create vmnet --fixed-range-v4=
> > 
> > 10.0.0.0/24--bridge-
> > 
> > > >> > > >>> interface=br100 --multi-host=T
> > > >> > > >>> 
> > > >> > > >>> just does nothing until canceled. I tried on the controller
> > 
> > and
> > 
> > > >> > > >>> compute1
> > > >> > > >>> node.
> > > >> > > >>> 
> > > >> > > >>> nova-api.log shows:
> > > >> > > >>> 
> > > >> > > >>> ERROR nova.openstack.common.rpc.common AMQP server on
> > > >> 
> > > >> controller:5672
> > > >> 
> > > >> > > >>> is
> > > >> > > >>> unreachable: Socket closed. Trying again in 1 seconds.
> > > >> > > >>> INFO nova.openstack.common.rpc.common Reconnecting to AMQP
> > 
> > server
> > 
> > > >> on
> > > >> 
> > > >> > > >>> controller:5672
> > > >> > > >>> 
> > > >> > > >>> controller:~# netstat -lpn | grep beam
> > > >> > > >>> tcp 0 0 0.0.0.0:36595 0.0.0.0:*
> > > >> > > >>> LISTEN
> > > >> > > >>> 1045/beam
> > > >> > > >>> tcp6 0 0 :::5672 :::*
> > > >> > > >>> LISTEN
> > > >> > > >>> 1045/beam
> > > >> > > >>> 
> > > >> > > >>> It listens on this port, but IPv6. IMO on linux listing on an
> > > >> > > >>> IPv6
> > > >> > > >>> port
> > > >> > > >>> also means listening on the same v4 port.
> > > >> > > >>> 
> > > >> > > >>> What's wrong with the installation?
> > > >> 
> > > >> _______________________________________________
> > > >> Mailing list:
> > > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> > > >> Post to     : openstack at lists.openstack.org
> > > >> Unsubscribe :
> > > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack





More information about the Openstack mailing list