<div dir="ltr">Josh,<div><br></div><div>Certainly in our case the the monitor hosts (in addition to IPs) would have made a difference.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, May 7, 2015 at 3:21 PM, Josh Durgin <span dir="ltr"><<a href="mailto:jdurgin@redhat.com" target="_blank">jdurgin@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hey folks, thanks for filing a bug for this:<br>
<br>
<a href="https://bugs.launchpad.net/cinder/+bug/1452641" target="_blank">https://bugs.launchpad.net/cinder/+bug/1452641</a><br>
<br>
Nova stores the volume connection info in its db, so updating that<br>
would be a workaround to allow restart/migration of vms to work.<br>
Otherwise running vms shouldn't be affected, since they'll notice any<br>
new or deleted monitors through their existing connection to the<br>
monitor cluster.<br>
<br>
Perhaps the most general way to fix this would be for cinder to return<br>
any monitor hosts listed in ceph.conf (as they are listed, so they may<br>
be hostnames or ips) in addition to the ips from the current monmap<br>
(the current behavior).<br>
<br>
That way an out of date ceph.conf is less likely to cause problems,<br>
and multiple clusters could still be used with the same nova node.<br>
<br>
Josh<span class=""><br>
<br>
On 05/06/2015 12:46 PM, David Medberry wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
Hi Arne,<br>
<br>
We've had this EXACT same issue.<br>
<br>
I don't know of a way to force an update as you are basically pulling<br>
the rug out from under a running instance. I don't know if it is<br>
possible/feasible to update the virsh xml in place and then migrate to<br>
get it to actually use that data. (I think we tried that to no avail.)<br>
dumpxml=>massage cephmons=>import xml<br>
<br>
If you find a way, let me know, and that's part of the reason I'm<br>
replying so that I stay on this thread. NOTE: We did this on icehouse.<br>
Haven't tried since upgrading to Juno but I don't note any change<br>
therein that would mitigate this. So I'm guessing Liberty/post-Liberty<br>
for a real fix.<br>
<br>
<br>
<br>
On Wed, May 6, 2015 at 12:57 PM, Arne Wiebalck <<a href="mailto:arne.wiebalck@cern.ch" target="_blank">arne.wiebalck@cern.ch</a><br></span><div><div class="h5">
<mailto:<a href="mailto:arne.wiebalck@cern.ch" target="_blank">arne.wiebalck@cern.ch</a>>> wrote:<br>
<br>
Hi,<br>
<br>
As we swapped a fraction of our Ceph mon servers between the<br>
pre-production and production cluster<br>
— something we considered to be transparent as the Ceph config<br>
points to the mon alias—, we ended<br>
up in a situation where VMs with volumes attached were not able to<br>
boot (with a probability that matched<br>
the fraction of the servers moved between the Ceph instances).<br>
<br>
We found that the reason for this is the connection_info in<br>
block_device_mapping which contains the<br>
IP adresses of the mon servers as extracted by the rbd driver in<br>
initialize_connection() at the moment<br>
when the connection is established. From what we see, however, this<br>
information is not updated as long<br>
as the connection exists, and will hence be re-applied without<br>
checking even when the XML is recreated.<br>
<br>
The idea to extract the mon servers by IP from the mon map was<br>
probably to get all mon servers (rather<br>
than only one from a load-balancer or an alias), but while our<br>
current scenario may be special, we will face<br>
a similar problem the day the Ceph mons need to be replaced. And<br>
that makes it a more general issue.<br>
<br>
For our current problem:<br>
Is there a user-transparent way to force an update of that<br>
connection information? (Apart from fiddling<br>
with the database entries, of course.)<br>
<br>
For the general issue:<br>
Would it be possible to simply use the information from the<br>
ceph.conf file directly (an alias in our case)<br>
throughout the whole stack to avoid hard-coding IPs that will be<br>
obsolete one day?<br>
<br>
Thanks!<br>
Arne<br>
<br>
—<br>
Arne Wiebalck<br>
CERN IT<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br>
<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br></div></div>
<<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><span class=""><br>
<br>
<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</span></blockquote><div class="HOEnZb"><div class="h5">
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>