<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 8, 2021 at 2:27 AM Adam Zheng <<a href="mailto:adam.zheng@colorado.edu">adam.zheng@colorado.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="overflow-wrap: break-word;" lang="EN-US">
<div class="gmail-m_4596737978605523740WordSection1">
<p class="MsoNormal">Hello Alan,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thank you for the clarification and the pointers.<u></u><u></u></p>
<p class="MsoNormal">I did also previously find that only the cinder client had the az option, which appeared to work on backing up a volume from another az.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">However, is there a way to get this working from horizon? While I can certainly make pages for my users that they will need to use the cli to do backups I feel it is not very friendly on my part to do that.</p></div></div></blockquote><div><br></div><div>Adding [horizon] tag to involve that community.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="overflow-wrap: break-word;" lang="EN-US"><div class="gmail-m_4596737978605523740WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal">For now if there is not a way, I may leave the az for cinder on nova so backups/restores work in properly in horizon. I can control where the data is in ceph; was mainly just hoping to set this in openstack for aesthetic/clarity (ie which
datacenter they are saving their volumes) for users utilizing the horizon volumes interface.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">--<u></u><u></u></p>
<p class="MsoNormal">Adam<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-color:rgb(181,196,223) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">Alan Bishop <<a href="mailto:abishop@redhat.com" target="_blank">abishop@redhat.com</a>><br>
<b>Date: </b>Friday, February 5, 2021 at 1:49 PM<br>
<b>To: </b>Adam Zheng <<a href="mailto:adam.zheng@colorado.edu" target="_blank">adam.zheng@colorado.edu</a>><br>
<b>Cc: </b>"<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>" <<a href="mailto:openstack-discuss@lists.openstack.org" target="_blank">openstack-discuss@lists.openstack.org</a>><br>
<b>Subject: </b>Re: [ops][cinder][kolla-ansible] cinder-backup fails if source disk not in nova az<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Fri, Feb 5, 2021 at 10:00 AM Adam Zheng <<a href="mailto:adam.zheng@colorado.edu" target="_blank">adam.zheng@colorado.edu</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">Hello,</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier"> </span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">I’ve been trying to get availability zones defined for volumes.</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">Everything works fine if I leave the zone at “nova”, all volume types work and backups/snapshots also work.</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier"> </span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">ie:</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">+------------------+----------------------------+------+---------+-------+----------------------------+</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| Binary | Host | Zone | Status | State | Updated At |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">+------------------+----------------------------+------+---------+-------+----------------------------+</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-scheduler | cs-os-ctl-001 | nova | enabled | up | 2021-02-05T17:22:51.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-scheduler | cs-os-ctl-003 | nova | enabled | up | 2021-02-05T17:22:54.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-scheduler | cs-os-ctl-002 | nova | enabled | up | 2021-02-05T17:22:56.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-001@rbd-ceph-gp2 | nova | enabled | up | 2021-02-05T17:22:56.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-001@rbd-ceph-st1 | nova | enabled | up | 2021-02-05T17:22:54.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-002@rbd-ceph-gp2 | nova | enabled | up | 2021-02-05T17:22:50.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-003@rbd-ceph-gp2 | nova | enabled | up | 2021-02-05T17:22:55.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-002@rbd-ceph-st1 | nova | enabled | up | 2021-02-05T17:22:57.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-003@rbd-ceph-st1 | nova | enabled | up | 2021-02-05T17:22:54.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-backup | cs-os-ctl-002 | nova | enabled | up | 2021-02-05T17:22:56.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-backup | cs-os-ctl-001 | nova | enabled | up | 2021-02-05T17:22:53.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-backup | cs-os-ctl-003 | nova | enabled | up | 2021-02-05T17:22:58.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">+------------------+----------------------------+------+---------+-------+----------------------------+</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier"> </span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">However, if I apply the following changes:</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier"> </span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">cinder-api.conf</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990p1"><span class="gmail-m_4596737978605523740gmail-m996591260014596990s1"><span style="font-family:Courier">[DEFAULT]</span></span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990p1"><span class="gmail-m_4596737978605523740gmail-m996591260014596990s1"><span style="font-family:Courier">default_availability_zone = not-nova</span></span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990p1"><span class="gmail-m_4596737978605523740gmail-m996591260014596990s1"><span style="font-family:Courier">default_volume_type = ceph-gp2</span></span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990p1"><span style="font-family:Courier">allow_availability_zone_fallback=True</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier"> </span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">cinder-volume.conf</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990p1"><span class="gmail-m_4596737978605523740gmail-m996591260014596990s1"><span style="font-family:Courier">[rbd-ceph-gp2]</span></span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier"><…></span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990p1"><span class="gmail-m_4596737978605523740gmail-m996591260014596990s1"><span style="font-family:Courier">backend_availability_zone = not-nova</span></span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier"><…></span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier"> </span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">I’ll get the following</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">+------------------+----------------------------+----------+---------+-------+----------------------------+</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| Binary | Host | Zone | Status | State | Updated At |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">+------------------+----------------------------+----------+---------+-------+----------------------------+</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-scheduler | cs-os-ctl-001 | nova | enabled | up | 2021-02-05T17:22:51.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-scheduler | cs-os-ctl-003 | nova | enabled | up | 2021-02-05T17:22:54.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-scheduler | cs-os-ctl-002 | nova | enabled | up | 2021-02-05T17:22:56.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-001@rbd-ceph-gp2 | not-nova | enabled | up | 2021-02-05T17:22:56.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-001@rbd-ceph-st1 | nova | enabled | up | 2021-02-05T17:22:54.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-002@rbd-ceph-gp2 | not-nova | enabled | up | 2021-02-05T17:22:50.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-003@rbd-ceph-gp2 | not-nova | enabled | up | 2021-02-05T17:22:55.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-002@rbd-ceph-st1 | nova | enabled | up | 2021-02-05T17:22:57.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-volume | cs-os-ctl-003@rbd-ceph-st1 | nova | enabled | up | 2021-02-05T17:22:54.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-backup | cs-os-ctl-002 | nova | enabled | up | 2021-02-05T17:22:56.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-backup | cs-os-ctl-001 | nova | enabled | up | 2021-02-05T17:22:53.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">| cinder-backup | cs-os-ctl-003 | nova | enabled | up | 2021-02-05T17:22:58.000000 |</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">+------------------+----------------------------+----------+---------+-------+----------------------------+</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier"> </span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">At this point, creating new volumes still work and go into the expected ceph pools.</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">However, backups no longer work for the cinder-volume that is not nova.</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">In the above example, it still works fine for volumes that that were created with type “ceph-gp2” in az “nova”.</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">Does not work for volumes that were created with type “ceph-st1” in az “not-nova”. It fails immediately and goes into error state with reason “Service not found for creating
backup.”</span><u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal">Hi Adam,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Cinder's backup service has the ability to create backups of volumes in another AZ. The 'cinder' CLI supports this feature as of microversion 3.51. (bear in mind the 'openstack' client doesn't support microversions for the cinder (volume)
service, so you'll need to use the 'cinder' command.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Rather than repeat what I've written previously, I refer you to [1] for additional details.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">[1] <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1649845#c4" target="_blank">
https://bugzilla.redhat.com/show_bug.cgi?id=1649845#c4</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">One other thing to note is the corresponding "cinder backup-restore" command currently does not support restoring to a volume in another AZ, but there is a workaround. You can pre-create a new volume in the destination AZ, and use the ability
to restore a backup to a specific volume (which just happens to be in your desired AZ).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">There's also a patch [2] under review to enhance the cinder shell so that both backup and restore shell commands work the same way.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">[2] <a href="https://secure-web.cisco.com/1hrYHQDQ6kxUUd_07Px7CO6di6oB79D53zKGIR_vPJ2lcipWw9j2QSEcwa84nQqil-SPAa0_z_d_ODf0_giBzaP7pNf-J4jdtSpsgajD6hgf4PYDMYPjQdT7WLpyhutiJ4DkTay7NCZH0XRsWXIYHMKZorIEJu4OXtRlU9MIF1REa3VPvOpKKsb1aBveoCxfEkWvNbYeOBmmD-eGhS8upWeKnWnfgLUqo0N21HXN4qW0HyFTV372apM2H010yWmRxfVKi_SIx9IJjZnA6Qht6YaEI4FMLN0FscExnzIV5KU8xoy_k_uisPki4nkKj9JBp4yBvqA7t-AzX-7u2clp2w8ku4yrj4Ar_H4Rw94xoOnsz5qIkhu8j7hpApR5HXe40LS8JHeySxg4JFbyxN4KmGanmUxYgDMRnrU4OtJhsiDZfUKHUaNH6P_8HPJnhk_H7VoP0kmeSbO-BnoSJHz4i3w/https%3A%2F%2Freview.opendev.org%2Fc%2Fopenstack%2Fpython-cinderclient%2F%2B%2F762020" target="_blank">
https://review.opendev.org/c/openstack/python-cinderclient/+/762020</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Alan<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">I suspect I need to try to get another set of “cinder-backup” services running in the Zone “not-nova”, but cannot seem to figure out how.</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier"> </span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">I’ve scoured the docs on cinder.conf, and if I set default zones in cinder-backup (I’ve tried backend_availability_zone, default_availability_zone, and storage_availability_zone)
I cannot seem to get backups working if the disk it’s backing up is not in az “nova”. The cinder-backup service in volume service list will always show “nova” no matter what I put there.</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier"> </span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">Any advice would be appreciated.</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">OpenStack Victoria deployed via kolla-ansible</span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier"> </span><u></u><u></u></p>
<p class="gmail-m_4596737978605523740gmail-m996591260014596990msoplaintext"><span style="font-family:Courier">Thanks!</span><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote></div></div>