Open Stack

For completeness sake here's what I ended up doing to resolve my ongoing issues with second Cinder node:

we're dealing with RHEL6.x here with Folsom from EPEL.

For simplification of things both iptables and SELinux were dropped into permissive mode on both nodes (which I will be bringing back up right away)


Multi-node Cinder
=================

We will use 2 nodes:

* master

  * openstack-keystone
  * openstack-cinder-api
  * openstack-cinder-scheduler
  * openstack-cinder-volume
  * qpidd 
  * tgtd
  * ...

* slave

  * openstack-cinder-volume
  * tgtd

* Networking

  * 10.10.10.0/24 - Admin network

    * 10.10.10.1 - master
    * 10.10.10.2 - slave

Note
----

Master node also hosts keystone and other core OpenStack services.

Configuration
=============

Make sure you're configuring your OpenStack infrastructure without the use of localhost and 127.0.0.1 references.

Also make sure your tgt is configured appropriately (openstack-setup scripts will do it for you). For RHEL <6.4 ::

    if [ "$DISTRO_VER" -eq '6' ] ; then
      if [ "$DISTRO_REL" -le '3' ] ; then
        
        ## as per comment in /etc/tgt/conf.d/cinder.conf
        ## while not flexible or smart - it should do the trick
        grep -F 'include /etc/cinder/volumes/*' /etc/tgt/targets.conf || \
         sed -i '0 i ##OPENSTACK-SETUP: \
    include /etc/cinder/volumes/* \
    ' /etc/tgt/targets.conf
      fi

Master

------

/etc/cinder/cinder.conf
~~~~~~~~~~~~~~~~~~~~~~~

First, the contents::

    [DEFAULT]
    logdir = /var/log/cinder
    state_path = /var/lib/cinder
    lock_path = /var/lib/cinder/tmp
    volumes_dir = /etc/cinder/volumes

    pybasedir=/usr/lib/python2.6/site-packages

    # Directory where cinder binaries are installed (string value)
    bindir=/usr/bin

    my_ip = 10.10.10.1
    iscsi_helper = tgtadm
    iscsi_ip_prefix= 10.10.10.1
    iscsi_ip_address= 10.10.10.1
    sql_connection = mysql://cinder:cinder@localhost/cinder

    rpc_backend = cinder.openstack.common.rpc.impl_qpid
    rootwrap_config = /etc/cinder/rootwrap.conf
    auth_strategy = keystone

    [keystone_authtoken]
    admin_tenant_name = service
    admin_user = cinder
    admin_password = cinder
    auth_host = 10.10.10.1
    auth_port = 35357
    auth_protocol = http
    signing_dirname = /tmp/keystone-signing-cinder


/etc/cinder/api-paste.ini

~~~~~~~~~~~~~~~~~~~~~~~~~

We'll skip the boring part and look at filter:authtoken::

    ....
    [filter:authtoken]
    paste.filter_factory = keystone.middleware.auth_token:filter_factory
    service_protocol = http
    service_host = 10.10.10.1
    service_port = 5000

Slave
-----


/etc/cinder/cinder.conf
~~~~~~~~~~~~~~~~~~~~~~~

Content is slightly different from Master::

    [DEFAULT]
    ## See more options at: 
    ## https://github.com/openstack/cinder/blob/master/etc/cinder/cinder.conf.sample
    logdir = /var/log/cinder
    state_path = /var/lib/cinder
    lock_path = /var/lib/cinder/tmp
    volumes_dir = /etc/cinder/volumes

    pybasedir=/usr/lib/python2.6/site-packages

    # Directory where cinder binaries are installed (string value)
    bindir=/usr/bin

    my_ip = 10.10.10.2

    iscsi_helper = tgtadm
    iscsi_ip_prefix= 10.10.10.2
    iscsi_ip_address= 10.10.10.2
    sql_connection = mysql://cinder:cinder@10.10.10.1/cinder
    rpc_backend = cinder.openstack.common.rpc.impl_qpid

    # Qpid broker hostname (string value)
    qpid_hostname=10.10.10.1

    rootwrap_config = /etc/cinder/rootwrap.conf
    auth_strategy = keystone

    [keystone_authtoken]
    admin_tenant_name = service
    admin_user = cinder
    admin_password = cinder
    auth_host = 10.10.10.1
    auth_port = 35357
    auth_protocol = http
    signing_dirname = /tmp/keystone-signing-cinder

Note that we have defined qpid_hostname and our sql_connection now uses proper IP


/etc/cinder/api-paste.ini
~~~~~~~~~~~~~~~~~~~~~~~~~

this one is pretty much a carbon-copy of Master::

    ....
    [filter:authtoken]
    paste.filter_factory = keystone.middleware.auth_token:filter_factory
    service_protocol = http
    service_host = 10.10.10.1
    service_port = 5000



Looks like two things have helped: 

1. Using non-localhost URLs for endpoints and other configuration
2. adding "my_ip" and "qpid_hostname" to the config in addition to existing "iscsi_ip*" stuff.