<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
Hi Josh,
<div><br>
</div>
<div>In our case adding the monitor hostnames (alias) would have made only a slight difference:</div>
<div>as we moved the servers to another cluster, the client received an authorisation failure rather</div>
<div>than a connection failure and did not try to fail over to the next IP in the list. So, adding the</div>
<div>alias to list would have improved the chances to hit a good monitor, but it would not have</div>
<div>eliminated the problem. </div>
<div><br>
</div>
<div>I’m not sure storing IPs in the nova database is a good idea in gerenal. Replacing (not adding)</div>
<div>these by the hostnames is probably better. Another approach may be to generate this part of</div>
<div>connection_info (and hence the XML) dynamically from the local ceph.conf when the connection</div>
<div>is created. I think a mechanism like this is for instance used to select a free port for the vnc</div>
<div>console when the instance is started.</div>
<div><br>
</div>
<div>Cheers,</div>
<div> Arne</div>
<div><br>
</div>
<div>—</div>
<div>Arne Wiebalck</div>
<div>CERN IT</div>
<div><br>
</div>
<div><br>
<div>
<div>On 08 May 2015, at 05:37, David Medberry <<a href="mailto:openstack@medberry.net">openstack@medberry.net</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div dir="ltr">Josh,
<div><br>
</div>
<div>Certainly in our case the the monitor hosts (in addition to IPs) would have made a difference.</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Thu, May 7, 2015 at 3:21 PM, Josh Durgin <span dir="ltr">
<<a href="mailto:jdurgin@redhat.com" target="_blank">jdurgin@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hey folks, thanks for filing a bug for this:<br>
<br>
<a href="https://bugs.launchpad.net/cinder/+bug/1452641" target="_blank">https://bugs.launchpad.net/cinder/+bug/1452641</a><br>
<br>
Nova stores the volume connection info in its db, so updating that<br>
would be a workaround to allow restart/migration of vms to work.<br>
Otherwise running vms shouldn't be affected, since they'll notice any<br>
new or deleted monitors through their existing connection to the<br>
monitor cluster.<br>
<br>
Perhaps the most general way to fix this would be for cinder to return<br>
any monitor hosts listed in ceph.conf (as they are listed, so they may<br>
be hostnames or ips) in addition to the ips from the current monmap<br>
(the current behavior).<br>
<br>
That way an out of date ceph.conf is less likely to cause problems,<br>
and multiple clusters could still be used with the same nova node.<br>
<br>
Josh<span class=""><br>
<br>
On 05/06/2015 12:46 PM, David Medberry wrote:<br>
</span>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class="">Hi Arne,<br>
<br>
We've had this EXACT same issue.<br>
<br>
I don't know of a way to force an update as you are basically pulling<br>
the rug out from under a running instance. I don't know if it is<br>
possible/feasible to update the virsh xml in place and then migrate to<br>
get it to actually use that data. (I think we tried that to no avail.)<br>
dumpxml=>massage cephmons=>import xml<br>
<br>
If you find a way, let me know, and that's part of the reason I'm<br>
replying so that I stay on this thread. NOTE: We did this on icehouse.<br>
Haven't tried since upgrading to Juno but I don't note any change<br>
therein that would mitigate this. So I'm guessing Liberty/post-Liberty<br>
for a real fix.<br>
<br>
<br>
<br>
On Wed, May 6, 2015 at 12:57 PM, Arne Wiebalck <<a href="mailto:arne.wiebalck@cern.ch" target="_blank">arne.wiebalck@cern.ch</a><br>
</span>
<div>
<div class="h5"><mailto:<a href="mailto:arne.wiebalck@cern.ch" target="_blank">arne.wiebalck@cern.ch</a>>> wrote:<br>
<br>
Hi,<br>
<br>
As we swapped a fraction of our Ceph mon servers between the<br>
pre-production and production cluster<br>
— something we considered to be transparent as the Ceph config<br>
points to the mon alias—, we ended<br>
up in a situation where VMs with volumes attached were not able to<br>
boot (with a probability that matched<br>
the fraction of the servers moved between the Ceph instances).<br>
<br>
We found that the reason for this is the connection_info in<br>
block_device_mapping which contains the<br>
IP adresses of the mon servers as extracted by the rbd driver in<br>
initialize_connection() at the moment<br>
when the connection is established. From what we see, however, this<br>
information is not updated as long<br>
as the connection exists, and will hence be re-applied without<br>
checking even when the XML is recreated.<br>
<br>
The idea to extract the mon servers by IP from the mon map was<br>
probably to get all mon servers (rather<br>
than only one from a load-balancer or an alias), but while our<br>
current scenario may be special, we will face<br>
a similar problem the day the Ceph mons need to be replaced. And<br>
that makes it a more general issue.<br>
<br>
For our current problem:<br>
Is there a user-transparent way to force an update of that<br>
connection information? (Apart from fiddling<br>
with the database entries, of course.)<br>
<br>
For the general issue:<br>
Would it be possible to simply use the information from the<br>
ceph.conf file directly (an alias in our case)<br>
throughout the whole stack to avoid hard-coding IPs that will be<br>
obsolete one day?<br>
<br>
Thanks!<br>
Arne<br>
<br>
—<br>
Arne Wiebalck<br>
CERN IT<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br>
<a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
</div>
</div>
<<a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><span class=""><br>
<br>
<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</span></blockquote>
<div class="HOEnZb">
<div class="h5"><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" target="_blank">
OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org">OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
<br>
</div>
</body>
</html>