Open Stack

Josh,

Certainly in our case the the monitor hosts (in addition to IPs) would have
made a difference.

On Thu, May 7, 2015 at 3:21 PM, Josh Durgin <jdurgin at redhat.com> wrote:

> Hey folks, thanks for filing a bug for this:
>
> https://bugs.launchpad.net/cinder/+bug/1452641
>
> Nova stores the volume connection info in its db, so updating that
> would be a workaround to allow restart/migration of vms to work.
> Otherwise running vms shouldn't be affected, since they'll notice any
> new or deleted monitors through their existing connection to the
> monitor cluster.
>
> Perhaps the most general way to fix this would be for cinder to return
> any monitor hosts listed in ceph.conf (as they are listed, so they may
> be hostnames or ips) in addition to the ips from the current monmap
> (the current behavior).
>
> That way an out of date ceph.conf is less likely to cause problems,
> and multiple clusters could still be used with the same nova node.
>
> Josh
>
> On 05/06/2015 12:46 PM, David Medberry wrote:
>
>> Hi Arne,
>>
>> We've had this EXACT same issue.
>>
>> I don't know of a way to force an update as you are basically pulling
>> the rug out from under a running instance. I don't know if it is
>> possible/feasible to update the virsh xml in place and then migrate to
>> get it to actually use that data. (I think we tried that to no avail.)
>> dumpxml=>massage cephmons=>import xml
>>
>> If you find a way, let me know, and that's part of the reason I'm
>> replying so that I stay on this thread. NOTE: We did this on icehouse.
>> Haven't tried since upgrading to Juno but I don't note any change
>> therein that would mitigate this. So I'm guessing Liberty/post-Liberty
>> for a real fix.
>>
>>
>>
>> On Wed, May 6, 2015 at 12:57 PM, Arne Wiebalck <arne.wiebalck at cern.ch
>> <mailto:arne.wiebalck at cern.ch>> wrote:
>>
>>     Hi,
>>
>>     As we swapped a fraction of our Ceph mon servers between the
>>     pre-production and production cluster
>>     -- something we considered to be transparent as the Ceph config
>>     points to the mon alias--, we ended
>>     up in a situation where VMs with volumes attached were not able to
>>     boot (with a probability that matched
>>     the fraction of the servers moved between the Ceph instances).
>>
>>     We found that the reason for this is the connection_info in
>>     block_device_mapping which contains the
>>     IP adresses of the mon servers as extracted by the rbd driver in
>>     initialize_connection() at the moment
>>     when the connection is established. From what we see, however, this
>>     information is not updated as long
>>     as the connection exists, and will hence be re-applied without
>>     checking even when the XML is recreated.
>>
>>     The idea to extract the mon servers by IP from the mon map was
>>     probably to get all mon servers (rather
>>     than only one from a load-balancer or an alias), but while our
>>     current scenario may be special, we will face
>>     a similar problem the day the Ceph mons need to be replaced. And
>>     that makes it a more general issue.
>>
>>     For our current problem:
>>     Is there a user-transparent way to force an update of that
>>     connection information? (Apart from fiddling
>>     with the database entries, of course.)
>>
>>     For the general issue:
>>     Would it be possible to simply use the information from the
>>     ceph.conf file directly (an alias in our case)
>>     throughout the whole stack to avoid hard-coding IPs that will be
>>     obsolete one day?
>>
>>     Thanks!
>>       Arne
>>
>>     --
>>     Arne Wiebalck
>>     CERN IT
>>
>> __________________________________________________________________________
>>     OpenStack Development Mailing List (not for usage questions)
>>     Unsubscribe:
>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>> >
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150507/368f42cb/attachment.html>