Open Stack

Hey folks, thanks for filing a bug for this:

https://bugs.launchpad.net/cinder/+bug/1452641

Nova stores the volume connection info in its db, so updating that
would be a workaround to allow restart/migration of vms to work.
Otherwise running vms shouldn't be affected, since they'll notice any
new or deleted monitors through their existing connection to the
monitor cluster.

Perhaps the most general way to fix this would be for cinder to return
any monitor hosts listed in ceph.conf (as they are listed, so they may
be hostnames or ips) in addition to the ips from the current monmap
(the current behavior).

That way an out of date ceph.conf is less likely to cause problems,
and multiple clusters could still be used with the same nova node.

Josh

On 05/06/2015 12:46 PM, David Medberry wrote:
> Hi Arne,
>
> We've had this EXACT same issue.
>
> I don't know of a way to force an update as you are basically pulling
> the rug out from under a running instance. I don't know if it is
> possible/feasible to update the virsh xml in place and then migrate to
> get it to actually use that data. (I think we tried that to no avail.)
> dumpxml=>massage cephmons=>import xml
>
> If you find a way, let me know, and that's part of the reason I'm
> replying so that I stay on this thread. NOTE: We did this on icehouse.
> Haven't tried since upgrading to Juno but I don't note any change
> therein that would mitigate this. So I'm guessing Liberty/post-Liberty
> for a real fix.
>
>
>
> On Wed, May 6, 2015 at 12:57 PM, Arne Wiebalck <arne.wiebalck at cern.ch
> <mailto:arne.wiebalck at cern.ch>> wrote:
>
>     Hi,
>
>     As we swapped a fraction of our Ceph mon servers between the
>     pre-production and production cluster
>     — something we considered to be transparent as the Ceph config
>     points to the mon alias—, we ended
>     up in a situation where VMs with volumes attached were not able to
>     boot (with a probability that matched
>     the fraction of the servers moved between the Ceph instances).
>
>     We found that the reason for this is the connection_info in
>     block_device_mapping which contains the
>     IP adresses of the mon servers as extracted by the rbd driver in
>     initialize_connection() at the moment
>     when the connection is established. From what we see, however, this
>     information is not updated as long
>     as the connection exists, and will hence be re-applied without
>     checking even when the XML is recreated.
>
>     The idea to extract the mon servers by IP from the mon map was
>     probably to get all mon servers (rather
>     than only one from a load-balancer or an alias), but while our
>     current scenario may be special, we will face
>     a similar problem the day the Ceph mons need to be replaced. And
>     that makes it a more general issue.
>
>     For our current problem:
>     Is there a user-transparent way to force an update of that
>     connection information? (Apart from fiddling
>     with the database entries, of course.)
>
>     For the general issue:
>     Would it be possible to simply use the information from the
>     ceph.conf file directly (an alias in our case)
>     throughout the whole stack to avoid hard-coding IPs that will be
>     obsolete one day?
>
>     Thanks!
>       Arne
>
>     —
>     Arne Wiebalck
>     CERN IT
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>