Open Stack

Hi Josh,

In our case adding the monitor hostnames (alias) would have made only a slight difference:
as we moved the servers to another cluster, the client received an authorisation failure rather
than a connection failure and did not try to fail over to the next IP in the list. So, adding the
alias to list would have improved the chances to hit a good monitor, but it would not have
eliminated the problem.

I’m not sure storing IPs in the nova database is a good idea in gerenal. Replacing (not adding)
these by the hostnames is probably better. Another approach may be to generate this part of
connection_info (and hence the XML) dynamically from the local ceph.conf when the connection
is created. I think a mechanism like this is for instance used to select a free port for the vnc
console when the instance is started.

Cheers,
 Arne

—
Arne Wiebalck
CERN IT


On 08 May 2015, at 05:37, David Medberry <openstack at medberry.net<mailto:openstack at medberry.net>> wrote:

Josh,

Certainly in our case the the monitor hosts (in addition to IPs) would have made a difference.

On Thu, May 7, 2015 at 3:21 PM, Josh Durgin <jdurgin at redhat.com<mailto:jdurgin at redhat.com>> wrote:
Hey folks, thanks for filing a bug for this:

https://bugs.launchpad.net/cinder/+bug/1452641

Nova stores the volume connection info in its db, so updating that
would be a workaround to allow restart/migration of vms to work.
Otherwise running vms shouldn't be affected, since they'll notice any
new or deleted monitors through their existing connection to the
monitor cluster.

Perhaps the most general way to fix this would be for cinder to return
any monitor hosts listed in ceph.conf (as they are listed, so they may
be hostnames or ips) in addition to the ips from the current monmap
(the current behavior).

That way an out of date ceph.conf is less likely to cause problems,
and multiple clusters could still be used with the same nova node.

Josh

On 05/06/2015 12:46 PM, David Medberry wrote:
Hi Arne,

We've had this EXACT same issue.

I don't know of a way to force an update as you are basically pulling
the rug out from under a running instance. I don't know if it is
possible/feasible to update the virsh xml in place and then migrate to
get it to actually use that data. (I think we tried that to no avail.)
dumpxml=>massage cephmons=>import xml

If you find a way, let me know, and that's part of the reason I'm
replying so that I stay on this thread. NOTE: We did this on icehouse.
Haven't tried since upgrading to Juno but I don't note any change
therein that would mitigate this. So I'm guessing Liberty/post-Liberty
for a real fix.



On Wed, May 6, 2015 at 12:57 PM, Arne Wiebalck <arne.wiebalck at cern.ch<mailto:arne.wiebalck at cern.ch>
<mailto:arne.wiebalck at cern.ch<mailto:arne.wiebalck at cern.ch>>> wrote:

    Hi,

    As we swapped a fraction of our Ceph mon servers between the
    pre-production and production cluster
    — something we considered to be transparent as the Ceph config
    points to the mon alias—, we ended
    up in a situation where VMs with volumes attached were not able to
    boot (with a probability that matched
    the fraction of the servers moved between the Ceph instances).

    We found that the reason for this is the connection_info in
    block_device_mapping which contains the
    IP adresses of the mon servers as extracted by the rbd driver in
    initialize_connection() at the moment
    when the connection is established. From what we see, however, this
    information is not updated as long
    as the connection exists, and will hence be re-applied without
    checking even when the XML is recreated.

    The idea to extract the mon servers by IP from the mon map was
    probably to get all mon servers (rather
    than only one from a load-balancer or an alias), but while our
    current scenario may be special, we will face
    a similar problem the day the Ceph mons need to be replaced. And
    that makes it a more general issue.

    For our current problem:
    Is there a user-transparent way to force an update of that
    connection information? (Apart from fiddling
    with the database entries, of course.)

    For the general issue:
    Would it be possible to simply use the information from the
    ceph.conf file directly (an alias in our case)
    throughout the whole stack to avoid hard-coding IPs that will be
    obsolete one day?

    Thanks!
      Arne

    —
    Arne Wiebalck
    CERN IT
    __________________________________________________________________________
    OpenStack Development Mailing List (not for usage questions)
    Unsubscribe:
    OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe>
    <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe>>
    http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev




__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org<mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150508/95bcf349/attachment.html>