<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 5, 2021 at 10:00 AM Adam Zheng <<a href="mailto:adam.zheng@colorado.edu">adam.zheng@colorado.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" lang="EN-US">
<div class="gmail-m_996591260014596990WordSection1">
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">Hello,<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><u></u> <u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">I’ve been trying to get availability zones defined for volumes.<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">Everything works fine if I leave the zone at “nova”, all volume types work and backups/snapshots also work.<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><u></u> <u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">ie:<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">+------------------+----------------------------+------+---------+-------+----------------------------+<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| Binary | Host | Zone | Status | State | Updated At |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">+------------------+----------------------------+------+---------+-------+----------------------------+<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-scheduler | cs-os-ctl-001 | nova | enabled | up | 2021-02-05T17:22:51.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-scheduler | cs-os-ctl-003 | nova | enabled | up | 2021-02-05T17:22:54.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-scheduler | cs-os-ctl-002 | nova | enabled | up | 2021-02-05T17:22:56.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-001@rbd-ceph-gp2 | nova | enabled | up | 2021-02-05T17:22:56.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-001@rbd-ceph-st1 | nova | enabled | up | 2021-02-05T17:22:54.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-002@rbd-ceph-gp2 | nova | enabled | up | 2021-02-05T17:22:50.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-003@rbd-ceph-gp2 | nova | enabled | up | 2021-02-05T17:22:55.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-002@rbd-ceph-st1 | nova | enabled | up | 2021-02-05T17:22:57.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-003@rbd-ceph-st1 | nova | enabled | up | 2021-02-05T17:22:54.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-backup | cs-os-ctl-002 | nova | enabled | up | 2021-02-05T17:22:56.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-backup | cs-os-ctl-001 | nova | enabled | up | 2021-02-05T17:22:53.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-backup | cs-os-ctl-003 | nova | enabled | up | 2021-02-05T17:22:58.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">+------------------+----------------------------+------+---------+-------+----------------------------+<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><u></u> <u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">However, if I apply the following changes:<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><u></u> <u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">cinder-api.conf<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990p1"><span class="gmail-m_996591260014596990s1"><span style="font-size:11pt;font-family:Courier">[DEFAULT]</span></span><span style="font-size:11pt;font-family:Courier"><u></u><u></u></span></p>
<p class="gmail-m_996591260014596990p1"><span class="gmail-m_996591260014596990s1"><span style="font-size:11pt;font-family:Courier">default_availability_zone = not-nova</span></span><span style="font-size:11pt;font-family:Courier"><u></u><u></u></span></p>
<p class="gmail-m_996591260014596990p1"><span class="gmail-m_996591260014596990s1"><span style="font-size:11pt;font-family:Courier">default_volume_type = ceph-gp2<u></u><u></u></span></span></p>
<p class="gmail-m_996591260014596990p1"><span style="font-size:11pt;font-family:Courier">allow_availability_zone_fallback=True<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><u></u> <u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">cinder-volume.conf<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990p1"><span class="gmail-m_996591260014596990s1"><span style="font-size:11pt;font-family:Courier">[rbd-ceph-gp2]</span></span><span style="font-size:11pt;font-family:Courier"><u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><…><u></u><u></u></span></p>
<p class="gmail-m_996591260014596990p1"><span class="gmail-m_996591260014596990s1"><span style="font-size:11pt;font-family:Courier">backend_availability_zone = not-nova</span></span><span style="font-size:11pt;font-family:Courier"><u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><…><u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><u></u> <u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">I’ll get the following<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">+------------------+----------------------------+----------+---------+-------+----------------------------+<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| Binary | Host | Zone | Status | State | Updated At |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">+------------------+----------------------------+----------+---------+-------+----------------------------+<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-scheduler | cs-os-ctl-001 | nova | enabled | up | 2021-02-05T17:22:51.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-scheduler | cs-os-ctl-003 | nova | enabled | up | 2021-02-05T17:22:54.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-scheduler | cs-os-ctl-002 | nova | enabled | up | 2021-02-05T17:22:56.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-001@rbd-ceph-gp2 | not-nova | enabled | up | 2021-02-05T17:22:56.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-001@rbd-ceph-st1 | nova | enabled | up | 2021-02-05T17:22:54.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-002@rbd-ceph-gp2 | not-nova | enabled | up | 2021-02-05T17:22:50.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-003@rbd-ceph-gp2 | not-nova | enabled | up | 2021-02-05T17:22:55.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-002@rbd-ceph-st1 | nova | enabled | up | 2021-02-05T17:22:57.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-003@rbd-ceph-st1 | nova | enabled | up | 2021-02-05T17:22:54.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-backup | cs-os-ctl-002 | nova | enabled | up | 2021-02-05T17:22:56.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-backup | cs-os-ctl-001 | nova | enabled | up | 2021-02-05T17:22:53.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">| cinder-backup | cs-os-ctl-003 | nova | enabled | up | 2021-02-05T17:22:58.000000 |<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">+------------------+----------------------------+----------+---------+-------+----------------------------+<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><u></u> <u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">At this point, creating new volumes still work and go into the expected ceph pools.<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">However, backups no longer work for the cinder-volume that is not nova.<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">In the above example, it still works fine for volumes that that were created with type “ceph-gp2” in az “nova”.<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">Does not work for volumes that were created with type “ceph-st1” in az “not-nova”. It fails immediately and goes into error state with reason “Service not found for creating backup.”</span></p></div></div></blockquote><div>Hi Adam,</div><div><br></div><div>Cinder's backup service has the ability to create backups of volumes in another AZ. The 'cinder' CLI supports this feature as of microversion 3.51. (bear in mind the 'openstack' client doesn't support microversions for the cinder (volume) service, so you'll need to use the 'cinder' command.</div><div><br></div><div>Rather than repeat what I've written previously, I refer you to [1] for additional details.</div><div><br></div><div>[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1649845#c4">https://bugzilla.redhat.com/show_bug.cgi?id=1649845#c4</a></div><div><br></div><div>One other thing to note is the corresponding "cinder backup-restore" command currently does not support restoring to a volume in another AZ, but there is a workaround. You can pre-create a new volume in the destination AZ, and use the ability to restore a backup to a specific volume (which just happens to be in your desired AZ).</div><div><br></div><div>There's also a patch [2] under review to enhance the cinder shell so that both backup and restore shell commands work the same way.</div><div><br></div><div>[2] <a href="https://review.opendev.org/c/openstack/python-cinderclient/+/762020">https://review.opendev.org/c/openstack/python-cinderclient/+/762020</a></div><div><br></div><div>Alan<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" lang="EN-US"><div class="gmail-m_996591260014596990WordSection1"><p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">I suspect I need to try to get another set of “cinder-backup” services running in the Zone “not-nova”, but cannot seem to figure out how.<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><u></u> <u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">I’ve scoured the docs on cinder.conf, and if I set default zones in cinder-backup (I’ve tried backend_availability_zone, default_availability_zone, and storage_availability_zone) I cannot seem to get
backups working if the disk it’s backing up is not in az “nova”. The cinder-backup service in volume service list will always show “nova” no matter what I put there.<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><u></u> <u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">Any advice would be appreciated.<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">OpenStack Victoria deployed via kolla-ansible<u></u><u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier"><u></u> <u></u></span></p>
<p class="gmail-m_996591260014596990MsoPlainText"><span style="font-family:Courier">Thanks!<u></u><u></u></span></p>
</div>
</div>
</blockquote></div></div>