Open Stack

Hi all,

I am testing Havana with ZeroMQ but I'm unable to make it work.

First of all, I have a few questions:

* I gather that the nova-rpc-zmq-receiver *must* run on *all* nodes
  (including compute nodes), is that correct?
* the nova-rpc-zmq-receiver is part (in Ubuntu) of the nova-scheduler
  package, should I install the package and disable nova-scheduler on
  the compute nodes? Is there an init script available for it or
  should I create my own?
* How the communication works: the nova-compute services talk to the
  nova-rpc-zmq-receiver via `tcp://` and the services on the same node
  as the nova-rpc-zmq-receiver talk using `ipc://`?

I am currently using two nodes: a controller node and a compute node.
On both nodes I added to nova.conf:

    rpc_zmq_bind_address = *
    rpc_zmq_contexts = 1
    rpc_zmq_host = cloud2.gc3
    rpc_zmq_ipc_dir = /var/run/openstack
    rpc_zmq_matchmaker =
nova.openstack.common.rpc.matchmaker_ring.MatchMakerRing
    rpc_zmq_port = 9501

started nova-rpc-zmq-receiver with:

    nova-rpc-zmq-receiver --config-file /etc/nova/nova.conf

and created a /etc/oslo/matchmaker_ring.json file containing:

    {
      "scheduler": ["cloud2"],
      "conductor": ["cloud2"],
      "cert": ["cloud2"],
      "consoleauth": ["cloud2"],
      "network": ["cloud2"],
      "compute": ["node-08-01-01"]

    }

where `cloud2` is my controller and `node-08-01-01` is my compute node.

However, when I run `nova service-list` (or `nova-manage service
list`) I don't see the compute node:

    +------------------+---------------+----------+---------+-------+----------------------------+-----------------+
    | Binary           | Host          | Zone     | Status  | State |
Updated_at                 | Disabled Reason |
    +------------------+---------------+----------+---------+-------+----------------------------+-----------------+
    | nova-consoleauth | cloud2        | internal | enabled | up    |
2014-03-25T15:23:49.000000 | None            |
    | nova-cert        | cloud2        | internal | enabled | up    |
2014-03-25T15:24:13.000000 | None            |
    | nova-scheduler   | cloud2        | internal | enabled | up    |
2014-03-25T15:23:36.000000 | None            |
    | nova-conductor   | cloud2        | internal | enabled | up    |
2014-03-25T15:23:56.000000 | None            |
    +------------------+---------------+----------+---------+-------+----------------------------+-----------------+

When I start the nova-compute service I see only the following lines
in the nova-compute.log:

    2014-03-25 16:27:05.158 7791 DEBUG
nova.openstack.common.rpc.common
[req-9bf7fc72-6419-4b62-aaa9-b1a7fac964ac None None] Sending
message(s) to: [(u'conductor.cloud2', u'cloud2')] _multi_send
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:725
    2014-03-25 16:27:05.159 7791 DEBUG
nova.openstack.common.rpc.common
[req-9bf7fc72-6419-4b62-aaa9-b1a7fac964ac None None] Creating payload
_call /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:650
    2014-03-25 16:27:05.159 7791 DEBUG
nova.openstack.common.rpc.common
[req-9bf7fc72-6419-4b62-aaa9-b1a7fac964ac None None] Creating queue
socket for reply waiter _call
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:663
    2014-03-25 16:27:05.162 7791 DEBUG
nova.openstack.common.rpc.common
[req-9bf7fc72-6419-4b62-aaa9-b1a7fac964ac None None] Subscribing to
0a9babcb7ba14b1180621583031a4223 subscribe
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:158
    2014-03-25 16:27:05.162 7791 DEBUG
nova.openstack.common.rpc.common
[req-9bf7fc72-6419-4b62-aaa9-b1a7fac964ac None None] Connecting to
ipc:///var/run/openstack/zmq_topic_zmq_replies.node-08-01-01.gc3 with
SUB __init__ /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:136
    2014-03-25 16:27:05.162 7791 DEBUG
nova.openstack.common.rpc.common
[req-9bf7fc72-6419-4b62-aaa9-b1a7fac964ac None None] -> Subscribed to
0a9babcb7ba14b1180621583031a4223 __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:137
    2014-03-25 16:27:05.162 7791 DEBUG
nova.openstack.common.rpc.common
[req-9bf7fc72-6419-4b62-aaa9-b1a7fac964ac None None] -> bind: False
__init__ /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:138
    2014-03-25 16:27:05.163 7791 DEBUG
nova.openstack.common.rpc.common
[req-9bf7fc72-6419-4b62-aaa9-b1a7fac964ac None None] Sending cast
_call /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:676
    2014-03-25 16:27:05.163 7791 DEBUG
nova.openstack.common.rpc.common
[req-9bf7fc72-6419-4b62-aaa9-b1a7fac964ac None None] Connecting to
tcp://cloud2:9501 with PUSH __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:136
    2014-03-25 16:27:05.163 7791 DEBUG
nova.openstack.common.rpc.common
[req-9bf7fc72-6419-4b62-aaa9-b1a7fac964ac None None] -> Subscribed to
None __init__ /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:137
    2014-03-25 16:27:05.164 7791 DEBUG
nova.openstack.common.rpc.common
[req-9bf7fc72-6419-4b62-aaa9-b1a7fac964ac None None] -> bind: False
__init__ /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:138
    2014-03-25 16:27:05.164 7791 DEBUG
nova.openstack.common.rpc.common
[req-9bf7fc72-6419-4b62-aaa9-b1a7fac964ac None None] Cast sent;
Waiting reply _call
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:679


and when I try to start a VM it stays in `BUILD/scheduling` state. On
the log of the compute node (verbose=True, debug=True) I don't see
anything, while the nova-{scheduler,conductor,cert}.log files only
contain the following line, repeated over and over again:

    2014-03-25 16:28:57.039 31938 DEBUG
nova.openstack.common.loopingcall [-] Dynamic looping call sleeping
for 60.00 seconds _inner
/usr/lib/python2.7/dist-packages/nova/openstack/common/loopingcall.py:132

The nova-rpc-zmq-receiver.log contains:

    2014-03-24 14:42:32.359 30163 INFO
nova.openstack.common.rpc.common [-] Registering reactor
    2014-03-24 14:42:32.361 30163 DEBUG
nova.openstack.common.rpc.common [-] Connecting to tcp://*:9501 with
PULL __init__ /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:136
    2014-03-24 14:42:32.361 30163 DEBUG
nova.openstack.common.rpc.common [-] -> Subscribed to None __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:137
    2014-03-24 14:42:32.362 30163 DEBUG
nova.openstack.common.rpc.common [-] -> bind: True __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:138
    2014-03-24 14:42:32.362 30163 INFO
nova.openstack.common.rpc.common [-] In reactor registered
    2014-03-24 14:42:32.362 30163 INFO
nova.openstack.common.rpc.common [-] Consuming socket
    2014-03-24 14:42:32.401 30163 INFO
nova.openstack.common.rpc.common [-] Creating proxy for topic:
q-plugin.localhost
    2014-03-24 14:42:32.401 30163 DEBUG
nova.openstack.common.rpc.common [-] Connecting to
ipc:///var/run/openstack/zmq_topic_q-plugin.localhost with PUSH
__init__ /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:136
    2014-03-24 14:42:32.401 30163 DEBUG
nova.openstack.common.rpc.common [-] -> Subscribed to None __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:137
    2014-03-24 14:42:32.402 30163 DEBUG
nova.openstack.common.rpc.common [-] -> bind: True __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:138
    2014-03-24 14:42:45.891 30163 INFO
nova.openstack.common.rpc.common [-] Creating proxy for topic:
q-l3-plugin.localhost
    2014-03-24 14:42:45.891 30163 DEBUG
nova.openstack.common.rpc.common [-] Connecting to
ipc:///var/run/openstack/zmq_topic_q-l3-plugin.localhost with PUSH
__init__ /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:136
    2014-03-24 14:42:45.892 30163 DEBUG
nova.openstack.common.rpc.common [-] -> Subscribed to None __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:137
    2014-03-24 14:42:45.892 30163 DEBUG
nova.openstack.common.rpc.common [-] -> bind: True __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:138
    2014-03-24 14:43:05.093 30163 INFO
nova.openstack.common.rpc.common [-] Creating proxy for topic:
conductor.cloud2.gc3
    2014-03-24 14:43:05.094 30163 DEBUG
nova.openstack.common.rpc.common [-] Connecting to
ipc:///var/run/openstack/zmq_topic_conductor.cloud2.gc3 with PUSH
__init__ /usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:136
    2014-03-24 14:43:05.094 30163 DEBUG
nova.openstack.common.rpc.common [-] -> Subscribed to None __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:137
    2014-03-24 14:43:05.094 30163 DEBUG
nova.openstack.common.rpc.common [-] -> bind: True __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:138
    2014-03-24 14:44:57.760 30163 INFO
nova.openstack.common.rpc.common [-] Creating proxy for topic:
conductor.cloud2
    2014-03-24 14:44:57.761 30163 DEBUG
nova.openstack.common.rpc.common [-] Connecting to
ipc:///var/run/openstack/zmq_topic_conductor.cloud2 with PUSH __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:136
    2014-03-24 14:44:57.761 30163 DEBUG
nova.openstack.common.rpc.common [-] -> Subscribed to None __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:137
    2014-03-24 14:44:57.761 30163 DEBUG
nova.openstack.common.rpc.common [-] -> bind: True __init__
/usr/lib/python2.7/dist-packages/nova/openstack/common/rpc/impl_zmq.py:138


Thank you in advance to anyone willing to help me :)
Antonio

-- 
antonio.s.messina at gmail.com
antonio.messina at uzh.ch                     +41 (0)44 635 42 22
GC3: Grid Computing Competence Center      http://www.gc3.uzh.ch/
University of Zurich
Winterthurerstrasse 190
CH-8057 Zurich Switzerland