I tried this in the lab, setting backend_host to "api-int.qde4.ourdomain.com" and my new volumes end up with:

os-vol-host-attr:host | api-int.qde4.cloud.ourdomain.com@rbd-1#rbd-1

And they work fine, but volumes that existed before the change cannot be attached, detached nor deleted. They still have the old setting:

os-vol-host-attr:host | qde4-ctrl2.cloud.ourdomain.com@rbd-1#rbd-1

I tried migrating them to the new host with:

openstack volume migrate --host api-int.qde4.cloud.ourdomain.com@rbd-1#rbd-1 <id>

But the migration never finishes:

| name | albert_testvol2 |
| os-vol-host-attr:host | qde4-ctrl2.cloud.ourdomain.com@rbd-1#rbd-1 |
| os-vol-mig-status-attr:migstat | starting

What am I missing? How can I fix my existing volumes after changing the backend_host?

On Wednesday, August 30, 2023 at 02:59:00 AM EDT, Gorka Eguileor <geguileo@redhat.com> wrote:

On 29/08, Albert Braden wrote:
>  What does backend_host look like? Should it match my internal API URL, i.e. api-int.qde4.ourdomain.com?
>      On Tuesday, August 29, 2023 at 10:44:48 AM EDT, Gorka Eguileor <geguileo@redhat.com> wrote:
>

Hi,

If I remember correctly you can set it to anything you want, but for
convenience I would recommend setting it to the hostname that currently
has more volumes in your system.

Let's say you have 3 hosts:
  qde4-ctrl1.cloud.ourdomain.com
  qde4-ctrl2.cloud.ourdomain.com
  qde4-ctrl3.cloud.ourdomain.com

And you have 100 volumes on ctrl1, 20 on ctrl2, and 10 on ctrl3.

Then it would be best to set it to ctrl1:
[rbd-1]
backend_host = qde4-ctrl1.cloud.ourdomain.com

And then use the cinder-manage command to modify the other 2.

For your information the value you see as "os-vol-host-attr:host" when
seeing the detailed information of a volume is in the form of:
<HostName>@<BackendName>#<PoolName>

In your case:
  <HostName> = qde4-ctrl1.cloud.ourdomain.com
  <BackendName> = rbd-1
  <PoolName> = rbd-1                         

In the RBD case the poolname will always be the same as the backendname.

Cheers,

Gorka.

>  On 29/08, Albert Braden wrote:
> > We’re replacing controllers, and it takes a few hours to build the new controller. We’re following this procedure to remove the old controller: https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hosts.html
> >
> >  After that the cluster seems to run fine on 2 controllers, but approximately 1/3 of our volumes can’t be attached to a VM. When we look at those volumes, we see this:
> >
> > | os-vol-host-attr:host          | qde4-ctrl1.cloud.ourdomain.com@rbd-1#rbd-1                            |
> >
> > Ctrl1 is the controller that is being replaced. Is it possible to change the os-vol-host-attr on a volume? How can we work around this issue while we are replacing controllers? Do we need to disable the API for the duration of the replacement process, or is there a better way?
> >
>
> Hi,
>
> Assuming you are running cinder volume in Active-Passive mode (which I
> believe was the only way back in Train) then you should be hardcoding
> the host name in the cinder.conf file to avoid losing access to your
> volumes when the volume service starts in another host.
>
> This is done with the "backend_host" configuration option within the
> specific driver section in cinder.conf.
>
> As for how to change the value of all the volumes to the same host
> value, you can use the "cinder-manage" command:
>
>   cinder-manage volume update_host \
>     --currenthost <current host> \
>     --newhost <new host>
>
> Cheers,
> Gorka.
>
>
>