Open Stack

Hi,

As we swapped a fraction of our Ceph mon servers between the pre-production and production cluster
— something we considered to be transparent as the Ceph config points to the mon alias—, we ended
up in a situation where VMs with volumes attached were not able to boot (with a probability that matched
the fraction of the servers moved between the Ceph instances).

We found that the reason for this is the connection_info in block_device_mapping which contains the
IP adresses of the mon servers as extracted by the rbd driver in initialize_connection() at the moment
when the connection is established. From what we see, however, this information is not updated as long
as the connection exists, and will hence be re-applied without checking even when the XML is recreated. 

The idea to extract the mon servers by IP from the mon map was probably to get all mon servers (rather
than only one from a load-balancer or an alias), but while our current scenario may be special, we will face
a similar problem the day the Ceph mons need to be replaced. And that makes it a more general issue.

For our current problem:
Is there a user-transparent way to force an update of that connection information? (Apart from fiddling
with the database entries, of course.)

For the general issue:
Would it be possible to simply use the information from the ceph.conf file directly (an alias in our case)
throughout the whole stack to avoid hard-coding IPs that will be obsolete one day?

Thanks!
 Arne

—
Arne Wiebalck
CERN IT