Open Stack

On 05/02, Alan Bishop wrote:
> On Fri, Feb 5, 2021 at 10:00 AM Adam Zheng <adam.zheng at colorado.edu> wrote:
>
> > Hello,
> >
> >
> >
> > I’ve been trying to get availability zones defined for volumes.
> >
> > Everything works fine if I leave the zone at “nova”, all volume types work
> > and backups/snapshots also work.
> >
> >
> >
> > ie:
> >
> >
> > +------------------+----------------------------+------+---------+-------+----------------------------+
> >
> > | Binary           | Host                       | Zone | Status  | State |
> > Updated At                 |
> >
> >
> > +------------------+----------------------------+------+---------+-------+----------------------------+
> >
> > | cinder-scheduler | cs-os-ctl-001              | nova | enabled | up    |
> > 2021-02-05T17:22:51.000000 |
> >
> > | cinder-scheduler | cs-os-ctl-003              | nova | enabled | up    |
> > 2021-02-05T17:22:54.000000 |
> >
> > | cinder-scheduler | cs-os-ctl-002              | nova | enabled | up    |
> > 2021-02-05T17:22:56.000000 |
> >
> > | cinder-volume    | cs-os-ctl-001 at rbd-ceph-gp2 | nova | enabled | up
> > | 2021-02-05T17:22:56.000000 |
> >
> > | cinder-volume    | cs-os-ctl-001 at rbd-ceph-st1 | nova | enabled | up
> > | 2021-02-05T17:22:54.000000 |
> >
> > | cinder-volume    | cs-os-ctl-002 at rbd-ceph-gp2 | nova | enabled | up
> > | 2021-02-05T17:22:50.000000 |
> >
> > | cinder-volume    | cs-os-ctl-003 at rbd-ceph-gp2 | nova | enabled | up
> > | 2021-02-05T17:22:55.000000 |
> >
> > | cinder-volume    | cs-os-ctl-002 at rbd-ceph-st1 | nova | enabled | up
> > | 2021-02-05T17:22:57.000000 |
> >
> > | cinder-volume    | cs-os-ctl-003 at rbd-ceph-st1 | nova | enabled | up
> > | 2021-02-05T17:22:54.000000 |
> >
> > | cinder-backup    | cs-os-ctl-002              | nova | enabled | up    |
> > 2021-02-05T17:22:56.000000 |
> >
> > | cinder-backup    | cs-os-ctl-001              | nova | enabled | up    |
> > 2021-02-05T17:22:53.000000 |
> >
> > | cinder-backup    | cs-os-ctl-003              | nova | enabled | up    |
> > 2021-02-05T17:22:58.000000 |
> >
> >
> > +------------------+----------------------------+------+---------+-------+----------------------------+
> >
> >
> >
> > However, if I apply the following changes:
> >
> >
> >
> > cinder-api.conf
> >
> > [DEFAULT]
> >
> > default_availability_zone = not-nova
> >
> > default_volume_type = ceph-gp2
> >
> > allow_availability_zone_fallback=True
> >
> >
> >
> > cinder-volume.conf
> >
> > [rbd-ceph-gp2]
> >
> > <…>
> >
> > backend_availability_zone = not-nova
> >
> > <…>
> >
> >
> >
> > I’ll get the following
> >
> >
> > +------------------+----------------------------+----------+---------+-------+----------------------------+
> >
> > | Binary           | Host                       | Zone     | Status  |
> > State | Updated At                 |
> >
> >
> > +------------------+----------------------------+----------+---------+-------+----------------------------+
> >
> > | cinder-scheduler | cs-os-ctl-001              | nova     | enabled |
> > up    | 2021-02-05T17:22:51.000000 |
> >
> > | cinder-scheduler | cs-os-ctl-003              | nova     | enabled |
> > up    | 2021-02-05T17:22:54.000000 |
> >
> > | cinder-scheduler | cs-os-ctl-002              | nova     | enabled |
> > up    | 2021-02-05T17:22:56.000000 |
> >
> > | cinder-volume    | cs-os-ctl-001 at rbd-ceph-gp2 | not-nova | enabled |
> > up    | 2021-02-05T17:22:56.000000 |
> >
> > | cinder-volume    | cs-os-ctl-001 at rbd-ceph-st1 | nova     | enabled |
> > up    | 2021-02-05T17:22:54.000000 |
> >
> > | cinder-volume    | cs-os-ctl-002 at rbd-ceph-gp2 | not-nova | enabled |
> > up    | 2021-02-05T17:22:50.000000 |
> >
> > | cinder-volume    | cs-os-ctl-003 at rbd-ceph-gp2 | not-nova | enabled |
> > up    | 2021-02-05T17:22:55.000000 |
> >
> > | cinder-volume    | cs-os-ctl-002 at rbd-ceph-st1 | nova     | enabled |
> > up    | 2021-02-05T17:22:57.000000 |
> >
> > | cinder-volume    | cs-os-ctl-003 at rbd-ceph-st1 | nova     | enabled |
> > up    | 2021-02-05T17:22:54.000000 |
> >
> > | cinder-backup    | cs-os-ctl-002              | nova     | enabled |
> > up    | 2021-02-05T17:22:56.000000 |
> >
> > | cinder-backup    | cs-os-ctl-001              | nova     | enabled |
> > up    | 2021-02-05T17:22:53.000000 |
> >
> > | cinder-backup    | cs-os-ctl-003              | nova     | enabled |
> > up    | 2021-02-05T17:22:58.000000 |
> >
> >
> > +------------------+----------------------------+----------+---------+-------+----------------------------+
> >
> >
> >
> > At this point, creating new volumes still work and go into the expected
> > ceph pools.
> >
> > However, backups no longer work for the cinder-volume that is not nova.
> >
> > In the above example, it still works fine for volumes that that were
> > created with type “ceph-gp2” in az “nova”.
> >
> > Does not work for volumes that were created with type “ceph-st1” in az
> > “not-nova”. It fails immediately and goes into error state with reason
> > “Service not found for creating backup.”
> >
> Hi Adam,
>
> Cinder's backup service has the ability to create backups of volumes in
> another AZ. The 'cinder' CLI supports this feature as of microversion 3.51.
> (bear in mind the 'openstack' client doesn't support microversions for the
> cinder (volume) service, so you'll need to use the 'cinder' command.
>
> Rather than repeat what I've written previously, I refer you to [1] for
> additional details.
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1649845#c4
>
> One other thing to note is the corresponding "cinder backup-restore"
> command currently does not support restoring to a volume in another AZ, but
> there is a workaround. You can pre-create a new volume in the destination
> AZ, and use the ability to restore a backup to a specific volume (which
> just happens to be in your desired AZ).
>

Hi,

Another way to restore a volume to a different AZ is to use the create
volume API with microversion 3.47 (instead of the backup-restore one)
where we can specify a backup id as the source for the volume, like we
do with snapshots and other volumes, and also specify the volume's AZ.

Cheers,
Gorka.

> There's also a patch [2] under review to enhance the cinder shell so that
> both backup and restore shell commands work the same way.
>
> [2] https://review.opendev.org/c/openstack/python-cinderclient/+/762020
>
> Alan
>
>
> > I suspect I need to try to get another set of “cinder-backup” services
> > running in the Zone “not-nova”, but cannot seem to figure out how.
> >
> >
> >
> > I’ve scoured the docs on cinder.conf, and if I set default zones in
> > cinder-backup (I’ve tried backend_availability_zone,
> > default_availability_zone, and storage_availability_zone) I cannot seem to
> > get backups working if the disk it’s backing up is not in az “nova”. The
> > cinder-backup service in volume service list will always show “nova” no
> > matter what I put there.
> >
> >
> >
> > Any advice would be appreciated.
> >
> > OpenStack Victoria deployed via kolla-ansible
> >
> >
> >
> > Thanks!
> >