<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Just to add to the cornucopia of configuration methods for cinder backends. We use a variation of this for active/active controller setup which seems to work (on pure, ceph rbd and nimble). We use both backend_host and volume_backend_name so any of our controllers
can action a request for a given volume. We've not had any issues with this setup since ussuri (we're on Zed / Yoga now).</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
eg:</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
[rbd]</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
backend_host = ceph-nvme<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
volume_backend_name = high-performance<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
...</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
[pure]</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted3">
backend_host = pure</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted3">
<span style="display: inline !important; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="ContentPasted4">volume_backend_name = high-performance</span><br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted3">
<span style="display: inline !important; color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);" class="ContentPasted4">...</span></div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
which leaves us with a set of the following:</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1 ContentPasted2">
$ openstack volume service list
<div class="ContentPasted2">+------------------+-------------------------+----------+---------+-------+----------------------------+</div>
<div class="ContentPasted2">| Binary | Host | Zone | Status | State | Updated At |</div>
<div class="ContentPasted2">+------------------+-------------------------+----------+---------+-------+----------------------------+</div>
<div class="ContentPasted2">| cinder-volume | nimble@nimble-az1 | gb-lon-1 | enabled | up | 2023-08-30T12:34:18.000000 |</div>
<div class="ContentPasted2">| cinder-volume | nimble@nimble-az2 | gb-lon-2 | enabled | up | 2023-08-30T12:34:14.000000 |</div>
<div class="ContentPasted2">| cinder-volume | nimble@nimble-az3 | gb-lon-3 | enabled | up | 2023-08-30T12:34:17.000000 |</div>
<div class="ContentPasted2">| cinder-volume | ceph-nvme@ceph-nvme-az3 | gb-lon-3 | enabled | up | 2023-08-30T12:34:14.000000 |</div>
<div class="ContentPasted2">| cinder-volume | pure@pure-az1 | gb-lon-1 | enabled | up | 2023-08-30T12:34:11.000000 |</div>
<div class="ContentPasted2">| cinder-volume | pure@pure-az2 | gb-lon-2 | enabled | up | 2023-08-30T12:34:11.000000 |</div>
<div class="ContentPasted2">+------------------+-------------------------+----------+---------+-------+----------------------------+</div>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Eugen Block <eblock@nde.ag><br>
<b>Sent:</b> 30 August 2023 11:36<br>
<b>To:</b> openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org><br>
<b>Subject:</b> Re: [kolla] [train] [cinder] Cinder issues during controller replacement</font>
<div> </div>
</div>
<div>CAUTION: This email originates from outside THG<br>
<br>
Just to share our config, we don't use "backend_host" but only "host" <br>
on all control nodes. Its value is the hostname (shortname) pointing <br>
to the virtual IP which migrates in case of a failure. We only use <br>
Ceph as storage backend with different volume types. The volume <br>
service list looks like this:<br>
<br>
controller02:~ # openstack volume service list<br>
+------------------+--------------------+------+---------+-------+----------------------------+<br>
| Binary | Host | Zone | Status | State | <br>
Updated At |<br>
+------------------+--------------------+------+---------+-------+----------------------------+<br>
| cinder-scheduler | controller | nova | enabled | up | <br>
2023-08-30T10:32:36.000000 |<br>
| cinder-backup | controller | nova | enabled | up | <br>
2023-08-30T10:32:34.000000 |<br>
| cinder-volume | controller@rbd | nova | enabled | up | <br>
2023-08-30T10:32:28.000000 |<br>
| cinder-volume | controller@rbd2 | nova | enabled | up | <br>
2023-08-30T10:32:36.000000 |<br>
| cinder-volume | controller@ceph-ec | nova | enabled | up | <br>
2023-08-30T10:32:28.000000 |<br>
+------------------+--------------------+------+---------+-------+----------------------------+<br>
<br>
We haven't seen any issue with this setup in years during failover.<br>
<br>
Zitat von Gorka Eguileor <geguileo@redhat.com>:<br>
<br>
> On 29/08, Albert Braden wrote:<br>
>> What does backend_host look like? Should it match my internal API <br>
>> URL, i.e. api-int.qde4.ourdomain.com?<br>
>> On Tuesday, August 29, 2023 at 10:44:48 AM EDT, Gorka Eguileor <br>
>> <geguileo@redhat.com> wrote:<br>
>><br>
><br>
> Hi,<br>
><br>
> If I remember correctly you can set it to anything you want, but for<br>
> convenience I would recommend setting it to the hostname that currently<br>
> has more volumes in your system.<br>
><br>
> Let's say you have 3 hosts:<br>
> qde4-ctrl1.cloud.ourdomain.com<br>
> qde4-ctrl2.cloud.ourdomain.com<br>
> qde4-ctrl3.cloud.ourdomain.com<br>
><br>
> And you have 100 volumes on ctrl1, 20 on ctrl2, and 10 on ctrl3.<br>
><br>
> Then it would be best to set it to ctrl1:<br>
> [rbd-1]<br>
> backend_host = qde4-ctrl1.cloud.ourdomain.com<br>
><br>
> And then use the cinder-manage command to modify the other 2.<br>
><br>
> For your information the value you see as "os-vol-host-attr:host" when<br>
> seeing the detailed information of a volume is in the form of:<br>
> <HostName>@<BackendName>#<PoolName><br>
><br>
> In your case:<br>
> <HostName> = qde4-ctrl1.cloud.ourdomain.com<br>
> <BackendName> = rbd-1<br>
> <PoolName> = rbd-1 <br>
><br>
> In the RBD case the poolname will always be the same as the backendname.<br>
><br>
> Cheers,<br>
> Gorka.<br>
><br>
>> On 29/08, Albert Braden wrote:<br>
>> > We’re replacing controllers, and it takes a few hours to build <br>
>> the new controller. We’re following this procedure to remove the <br>
>> old controller: <br>
>> <a href="https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hosts.html">
https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hosts.html</a><br>
>> ><br>
>> > After that the cluster seems to run fine on 2 controllers, but <br>
>> approximately 1/3 of our volumes can’t be attached to a VM. When we <br>
>> look at those volumes, we see this:<br>
>> ><br>
>> > | os-vol-host-attr:host | <br>
>> qde4-ctrl1.cloud.ourdomain.com@rbd-1#rbd-1 <br>
>> |<br>
>> ><br>
>> > Ctrl1 is the controller that is being replaced. Is it possible to <br>
>> change the os-vol-host-attr on a volume? How can we work around <br>
>> this issue while we are replacing controllers? Do we need to <br>
>> disable the API for the duration of the replacement process, or is <br>
>> there a better way?<br>
>> ><br>
>><br>
>> Hi,<br>
>><br>
>> Assuming you are running cinder volume in Active-Passive mode (which I<br>
>> believe was the only way back in Train) then you should be hardcoding<br>
>> the host name in the cinder.conf file to avoid losing access to your<br>
>> volumes when the volume service starts in another host.<br>
>><br>
>> This is done with the "backend_host" configuration option within the<br>
>> specific driver section in cinder.conf.<br>
>><br>
>> As for how to change the value of all the volumes to the same host<br>
>> value, you can use the "cinder-manage" command:<br>
>><br>
>> cinder-manage volume update_host \<br>
>> --currenthost <current host> \<br>
>> --newhost <new host><br>
>><br>
>> Cheers,<br>
>> Gorka.<br>
>><br>
>><br>
>><br>
<br>
<br>
<br>
</div>
</body>
</html>