This ended up being a cleanup issue in the management of the tmp file.  I will be submitting a PR for it.

On Wed, Feb 27, 2019 at 2:30 PM Tyler Bishop <tyler.bishop@beyondhosting.net> wrote:

Trying to deploy a new cluster using the bootstrapping for ceph but running into issues with the admin keys being incorrectly deployed.

Successful ansible deploy up until:

TASK [ceph : Getting ceph mgr keyring] ******************************************************************************************************************************

failed: [osctlr.home.visualbits.net -> osctlr.home.visualbits.net] (item=osctlr.home.visualbits.net) => {"changed": false, "item": "osctlr.home.visualbits.net", "msg": "Failed to call command: ['docker', 'exec', 'ceph_mon', 'ceph', '--format', 'json', 'auth', 'get-or-create', 'mgr.osctlr.home.visualbits.net', 'mds', 'allow *', 'mon', 'allow profile mgr', 'osd', 'allow *'] returncode: 1 output: stdout: \"\", stderr: \"[errno 1] error connecting to the cluster\n\""}

Errors in log from ceph:

TASK [ceph : Getting ceph mgr keyring] ******************************************************************************************************************************

failed: [osctlr.home.visualbits.net -> osctlr.home.visualbits.net] (item=osctlr.home.visualbits.net) => {"changed": false, "item": "osctlr.home.visualbits.net", "msg": "Failed to call command: ['docker', 'exec', 'ceph_mon', 'ceph', '--format', 'json', 'auth', 'get-or-create', 'mgr.osctlr.home.visualbits.net', 'mds', 'allow *', 'mon', 'allow profile mgr', 'osd', 'allow *'] returncode: 1 output: stdout: \"\", stderr: \"[errno 1] error connecting to the cluster\n\""}

keyrings look proper:

(openstack) [root@osctlr ~]# md5sum /etc/kolla/ceph-mon/ceph.client.admin.keyring

4658c01282c791bce9c75678df9e21c9  /etc/kolla/ceph-mon/ceph.client.admin.keyring

(openstack) [root@osctlr ~]# md5sum /var/lib/docker/volumes/ceph_mon_config/_data/ceph.client.admin.keyring

4658c01282c791bce9c75678df9e21c9  /var/lib/docker/volumes/ceph_mon_config/_data/ceph.client.admin.keyring

I've removed the docker container, volume and kolla config directories multple times with the same error.  I can't even run ceph status from the container bash itself.

Any ideas?