Issue with ceph and vault
Hi, I currently bumped into a situation in a cloud using both ceph and vault. While adding a new node to the cloud all seemed fine, until I noticed that on one node juju failed with the message: hook failed: "secrets-storage-relation-changed" Upon investigation it turned out that this node had a broken network configuration that was not picket up earlier. The problem was, that the vault instance could not be reached. This was fixed, but I could not un-wedge the error. A juju resolve on the node does not seem to have any effect. Using the debug-log option from juju I do see the following lines that do indicate the problem. INFO:vaultlocker.dmcrypt:LUKS formatting /dev/disk/by-dname/osd1-part1 using UUID:4448f9fa-d291-403e-9826-7318e57cf1a4 Cannot format device /dev/disk/by-dname/osd1-part1 which is still in use. vaultlocker: Command '['cryptsetup', '--batch-mode', '--uuid', '4448f9fa-d291-403e-9826-7318e57cf1a4', '--key-file', '-', 'luksFormat', '/dev/disk/by-dname/osd1-part1']' returned non-zero exit status 5 So the disk is in use. Which looks strange to me, as this machine could never complete the hooks as the vault was unreachable. I would like to get some advice on how to proceed. Regrds, Wouter
On 02/27/2019 10:35 PM, Wouter van Bommel wrote:
Hi,
I currently bumped into a situation in a cloud using both ceph and vault. While adding a new node to the cloud all seemed fine, until I noticed that on one node juju failed with the message: hook failed: "secrets-storage-relation-changed"
Upon investigation it turned out that this node had a broken network configuration that was not picket up earlier. The problem was, that the vault instance could not be reached.
This was fixed, but I could not un-wedge the error. A juju resolve on the node does not seem to have any effect.
Using the debug-log option from juju I do see the following lines that do indicate the problem.
INFO:vaultlocker.dmcrypt:LUKS formatting /dev/disk/by-dname/osd1-part1 using UUID:4448f9fa-d291-403e-9826-7318e57cf1a4 Cannot format device /dev/disk/by-dname/osd1-part1 which is still in use. vaultlocker: Command '['cryptsetup', '--batch-mode', '--uuid', '4448f9fa-d291-403e-9826-7318e57cf1a4', '--key-file', '-', 'luksFormat', '/dev/disk/by-dname/osd1-part1']' returned non-zero exit status 5
So the disk is in use. Which looks strange to me, as this machine could never complete the hooks as the vault was unreachable.
I would like to get some advice on how to proceed.
Hi Wouter, I'm not sure the OpenStack discuss mailing list is the best place for this question. I'm actually unsure what specific OpenStack component you're having an issue with. Is the error you show above an error in the nova-compute service? Or cinder-volume? Or something entirely different? I don't see mention of any OpenStack-specific things above. Best, -jay
participants (2)
-
Jay Pipes
-
Wouter van Bommel