<div dir="ltr"><div>Hi all, </div><div>I made some progress on this, but I am unsure how to make it better. Long story short: </div><div><ul><li>queens issue = snapshot is deleted from openstack but shouldnt be because the snapshot is unable to be deleted on the storage side</li><li>compared pike / queens / stein "nimble.py" and found all are different, with each newer version of openstack having additions in code</li><li>done some trial and error tests and decided to use the stein nimble.py and modify it </li><li>found the 'delete snapshot' section and found it is set to not halt on error</li><li>added 'raise' into the function and re-tested the 'delete snapshot' scenario </li><li>RESULT = now the snapshot is NOT deleted but goes "error deleting" instead :) </li></ul><div>So now after making that change, the snapshot is now in an unavailable status. I am looking as to how I can do something else other than make this snapshot go into an unavailable condition. Such as display a message while keeping the snapshot "available" because it can still be used <br></div><div><br></div></div><div>Short story long: </div><div><br></div>The "nimble.py" driver changes between pike,queens,stein versions (though within the file it has "driver version 4.0.1" on all). Pike has around 1700 lines. Queens has 1900 and Stein has 1910 approx. <div><br></div><div>I confirmed the issue with the driver by copying the nimble.py driver (and the other 2 files named nimble.pyc and nimble.pyo) from Pike into the Queens test env. to test if the snapshot still gets deleted under Queens or shows an error instead. The snapshot was not deleted and it goes error status as expected. </div><div>note: Initially, I only copied the text data from nimble.py and it appears as though the update to the text file was ignored. It looks to me like, openstack uses one of those .pyc or .pyo files instead. I googled on this and they are binaries that are used in some situations. If I make any changes to the nimble.py file then I need to re-generate those .pyc and .pyo files from the .py. <br></div><div><br></div><div>So what is happening here is; I want to try and delete a snapshot that has a clone. The expected outcome is the snapshot is not deleted in Openstack. Current experience is that Openstack deletes the snapshot from the volume snapshots, leaving the snapshot behind on the array storage side.</div><div><br></div><div>In the volume.log, I see the array sends back an error 409 with "has a clone" response. I managed to find which section is printing the error in the volume.log from the nimble.py driver file and so I edited the text section that gets printed and re-run the test. The volume.log now gets printed with the new text additions I added 'DELETE' and 'Response' words:</div><div><br></div><div>: NimbleAPIException: DELETE Failed to execute api snapshots/0464a5fd65d75fcfe1000000000000011d00001b1d: Response Error Code: 409 Message: Snapshot snapshot-4ee076ad-2e14-4d0d-bc20-64c17c741f8c for volume volume-7dd64cf1-1d56-4f21-a153-a8137b68c557 has a clone.<br></div><div><br></div><div>This is the python code where I made those changes: basically, because the error code is not 200 or 201 then it throws the error from what I understand. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> def delete(self, api):<br> url = self.uri + api<br> r = requests.delete(url, headers=self.headers, verify=self.verify)<br> if r.status_code != 201 and r.status_code != 200:<br> base = "DELETE Failed to execute api %(api)s: Response Error Code: %(code)s" % {<br> 'api': api,<br> 'code': r.status_code}<br> LOG.debug("Base error : %(base)s", {'base': base})<br> try:<br> msg = _("%(base)s Message: %(msg)s") % {<br> 'base': base,<br> 'msg': r.json()['messages'][1]['text']}<br> except IndexError:<br> msg = _("%(base)s Message: %(msg)s") % {<br> 'base': base,<br> 'msg': six.text_type(r.json())}<br> raise NimbleAPIException(msg)<br> return r.json()</blockquote><div><br></div><div><br></div><div>However, slightly before the above within nimble.py I think I have found the function that is causing the initial problem:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> def delete_snap(self, volume_name, snap_name):<br> snap_info = self.get_snap_info(snap_name, volume_name)<br> api = "snapshots/" + six.text_type(snap_info['id'])<br> try:<br> self.delete(api)<br> except NimbleAPIException as ex:<br> LOG.debug("delete snapshot exception: %s", ex)<br> if SM_OBJ_HAS_CLONE in six.text_type(ex):<br> # if snap has a clone log the error and continue ahead<br> LOG.warning('Snapshot %(snap)s : %(state)s',<br> {'snap': snap_name,<br> 'state': SM_OBJ_HAS_CLONE})<br> else:<br> raise</blockquote><div><br></div><div> SM_OBJ_HAS_CLONE is looking for "has a clone" and it's defined in the beginning of the file: "SM_OBJ_HAS_CLONE = "has a clone"" and I can see this in the log file "has a clone" as a response 409.<br></div><div><br></div><div>My problem is literally "
# if snap has a clone log the error and continue ahead" - it shouldnt be continuing, because by continuing it is deleting the snapshot on the Openstack side but is unable to do the same on the storage side because of the dependency issue. </div><div><br></div><div>So what I did next was to look into the different "delete volume" section for some help - because a similar condition can occur there -> to explain; if volume2 is a clone of volume1 then we can't delete volume1 until we first delete volume2. <br>What I notice is that there is a "raise" in that section at the end - I think I understand this to be throwing an exception to openstack. ie to cause an error condition. </div><div><br></div><div>Here's the delete volume section from the driver: </div><div><br></div><div><div style="color:rgb(212,212,212);background-color:rgb(30,30,30);font-family:Consolas,"Courier New",monospace;font-size:14px;line-height:19px;white-space:pre"><div><span style="color:rgb(86,156,214)">def</span> <span style="color:rgb(220,220,170)">delete_volume</span>(<span style="color:rgb(156,220,254)">self</span>, <span style="color:rgb(156,220,254)">volume</span>):</div><div> <span style="color:rgb(206,145,120)">"""Delete the specified volume."""</span></div><div> backup_snap_name, backup_vol_name = <span style="color:rgb(86,156,214)">self</span>.is_volume_backup_clone(volume)</div><div> eventlet.sleep(DEFAULT_SLEEP)</div><div> <span style="color:rgb(86,156,214)">self</span>.APIExecutor.online_vol(volume[<span style="color:rgb(206,145,120)">'name'</span>], <span style="color:rgb(86,156,214)">False</span>)</div><div> LOG.debug(<span style="color:rgb(206,145,120)">"Deleting volume </span><span style="color:rgb(86,156,214)">%(vol)s</span><span style="color:rgb(206,145,120)">"</span>, {<span style="color:rgb(206,145,120)">'vol'</span>: volume[<span style="color:rgb(206,145,120)">'name'</span>]})</div><br><div> <span style="color:rgb(220,220,170)">@utils.retry</span>(NimbleAPIException, <span style="color:rgb(156,220,254)">retries</span>=<span style="color:rgb(181,206,168)">3</span>)</div><div> <span style="color:rgb(86,156,214)">def</span> <span style="color:rgb(220,220,170)">_retry_remove_vol</span>(<span style="color:rgb(156,220,254)">volume</span>):</div><div> <span style="color:rgb(86,156,214)">self</span>.APIExecutor.delete_vol(volume[<span style="color:rgb(206,145,120)">'name'</span>])</div><div> <span style="color:rgb(197,134,192)">try</span>:</div><div> _retry_remove_vol(volume)</div><div> <span style="color:rgb(197,134,192)">except</span> NimbleAPIException <span style="color:rgb(197,134,192)">as</span> ex:</div><div> LOG.debug(<span style="color:rgb(206,145,120)">"delete volume exception: </span><span style="color:rgb(86,156,214)">%s</span><span style="color:rgb(206,145,120)">"</span>, ex)</div><div> <span style="color:rgb(197,134,192)">if</span> SM_OBJ_HAS_CLONE <span style="color:rgb(86,156,214)">in</span> six.text_type(ex):</div><div> LOG.warning(<span style="color:rgb(206,145,120)">'Volume </span><span style="color:rgb(86,156,214)">%(vol)s</span><span style="color:rgb(206,145,120)"> : </span><span style="color:rgb(86,156,214)">%(state)s</span><span style="color:rgb(206,145,120)">'</span>,</div><div> {<span style="color:rgb(206,145,120)">'vol'</span>: volume[<span style="color:rgb(206,145,120)">'name'</span>],</div><div> <span style="color:rgb(206,145,120)">'state'</span>: SM_OBJ_HAS_CLONE})</div><div> <span style="color:rgb(106,153,85)"># set the volume back to be online and raise busy exception</span></div><div> <span style="color:rgb(86,156,214)">self</span>.APIExecutor.online_vol(volume[<span style="color:rgb(206,145,120)">'name'</span>], <span style="color:rgb(86,156,214)">True</span>)</div><div> <span style="color:rgb(197,134,192)">raise</span> exception.VolumeIsBusy(<span style="color:rgb(156,220,254)">volume_name</span>=volume[<span style="color:rgb(206,145,120)">'name'</span>])</div><div> <span style="color:rgb(197,134,192)">raise</span></div></div></div><div><br></div><div><br></div><div>So with the above, I modified the delete snapshot section and put in a simple "raise" like this (highlighted in yellow)</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br> def delete_snap(self, volume_name, snap_name):<br> snap_info = self.get_snap_info(snap_name, volume_name)<br> api = "snapshots/" + six.text_type(snap_info['id'])<br> try:<br> self.delete(api)<br> except NimbleAPIException as ex:<br> LOG.debug("delete snapshot exception: %s", ex)<br> if SM_OBJ_HAS_CLONE in six.text_type(ex):<br> # if snap has a clone log the error and continue ahead<br> LOG.warning('Snapshot %(snap)s : %(state)s',<br> {'snap': snap_name,<br> 'state': SM_OBJ_HAS_CLONE})<br> <span style="background-color:rgb(255,255,0)">raise</span><br> else:<br> raise</blockquote><div><br></div><div><br></div><div>And now when I test, the snapshot is not deleted but it instead goes into ERROR-DELETING. It's not perfect but at least I can make the snapshot back to "available" from the admin section within Openstack. </div><div><br></div><div>Would anyone be able to if possible, give me some pointers how to accept this error but not cause the snapshot to go into "error" ? I think that I need to create a class?</div><div><br></div><div>regards</div><div><br></div><div><div><div dir="ltr" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><p style="background-image:initial;background-position:initial;background-repeat:initial"><a name="m_6330074280495010434_m_3038981133509348776_m_-1416124594182612983_SignatureSanitizer_SafeHtmlFilter__MailAutoSig"><b><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Tony Pearce</span></b></a><span style="font-size:9pt;font-family:Arial,sans-serif;color:black">
| </span><b><span style="font-family:Arial,sans-serif;color:rgb(0,112,192)"><span style="font-size:9pt">Senior Network Engineer /
Infrastructure Lead</span><span style="font-size:12px"><br></span></span></b><b><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(0,112,192)"><a href="https://www.cinglevue.com" target="_blank">Cinglevue International</a></span></b></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><span lang="EN-US" style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Email: </span><span style="font-family:Arial,sans-serif"><a href="mailto:tony.pearce@cinglevue.com" style="color:rgb(17,85,204);font-size:10pt" target="_blank">tony.pearce@cinglevue.com</a><span style="font-size:12.6667px"><br></span></span><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Web: </span><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(17,85,204)"><a href="http://www.cinglevue.com/" target="_blank">http://www.cinglevue.com</a></span><b><span style="font-size:9pt;font-family:Arial,sans-serif;color:rgb(38,66,120)"> </span></b></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><span lang="EN-US" style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><b><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Australia</span></b><span style="font-family:Arial,sans-serif"><font color="#000000"><span style="font-size:10pt"> </span></font><span style="font-size:12.6667px"><br></span></span><span style="color:black;font-family:Arial,sans-serif;font-size:10pt">1 Walsh Loop, Joondalup, WA 6027 Australia.</span></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><span lang="EN-US" style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Direct: </span><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(17,85,204)">+61 8 6202 0036</span><span style="font-size:10pt;font-family:Arial,sans-serif;color:black"> | </span><span lang="EN-US" style="font-size:9.5pt;font-family:Arial,sans-serif"></span><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Main: </span><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(17,85,204)">+61 8 6202 0024</span></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Note: This email and all attachments are the sole
property of Cinglevue International Pty Ltd. (or any of its subsidiary
entities), and the information contained herein must be considered
confidential, unless specified otherwise. If you are not the intended
recipient, you must not use or forward the information contained in these
documents. If you have received this message in error, please
delete the email and notify the sender.</span><span lang="EN-US" style="font-size:9.5pt;font-family:Arial,sans-serif"></span></p><p>
</p><p> </p></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, 18 Jan 2020 at 09:44, Tony Pearce <<a href="mailto:tony.pearce@cinglevue.com" target="_blank">tony.pearce@cinglevue.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto">Thank you. That really helps. <div dir="auto"><br></div><div dir="auto">I am going to diff the nimble.py files between Pike and Queens and see what's changed. </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 17 Jan 2020, 22:18 Alan Bishop, <<a href="mailto:abishop@redhat.com" target="_blank">abishop@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 17, 2020 at 2:01 AM Tony Pearce <<a href="mailto:tony.pearce@cinglevue.com" rel="noreferrer" target="_blank">tony.pearce@cinglevue.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Could anyone help by pointing me where to go to be able to dig into this issue further? </div><div><br></div><div>I have installed a test Openstack environment using RDO Packstack. I wanted to install the same version that I have in Production (Pike) but it's not listed in the CentOS repo via yum search. So I installed Queens. I am using nimble.py Cinder driver. Nimble Storage is a storage array accessed via iscsi from the Openstack host, and is controlled from Openstack by the driver and API. </div><div><b><br></b></div><div><div><b>What I expected to happen:</b></div><div>1. create an instance with volume (the volume is created on the storage array successfully and instance boots from it)</div><div>2. take a snapshot (snapshot taken on the volume on the array successfully)</div><div>3. create a new instance from the snapshot (the api tells the array to clone the snapshot into a new volume on the array and use that volume for the instance)</div><div>4. try and delete the snapshot<br>Expected Result - Openstack gives the user a message like "you're not allowed to do that".</div></div><div><br></div><div> Note: Step 3 above creates a child volume from the parent snapshot. It's impossible to delete the parent snapshot because IO READ is sent to that part of the original volume (as I understand it). <br></div><div><br></div><div><b>My production problem is this: </b></div><div>1. create an instance with volume (the volume is created on the storage array successfully)</div><div>2. take a snapshot (snapshot taken on the volume on the array successfully)</div><div>3. create a new instance from the snapshot (the api tells the array to clone the snapshot into a new volume on the array and use that volume for the instance)</div><div>4. try and delete the snapshot<br>Result - snapshot goes into error state and later, all Cinder operations fail such as new instance/create volume etc. until the correct service is restarted. Then everything works once again. </div><div><br></div><div><br></div><div>To troubleshoot the above, I installed the RDP Packstack Queens (because I couldnt get Pike). I tested the above and now, the result is the snapshot is successfully deleted from openstack but not deleted on the array. The log is below for reference. But I can see the in the log that the array sends back info to openstack saying the snapshot has a clone and the delete cannot be done because of that. Also response code 409. </div><div><br></div><div><b>Some info about why the problem with Pike started in the first place</b></div><div>1. Vendor is Nimble Storage which HPE purchased</div><div>2. HPE/Nimble have dropped support for openstack. Latest supported version is Queens and Nimble array version v4.x. The current Array version is v5.x. Nimble say there are no guarantees with openstack, the driver and the array version v5.x</div><div>3. I was previously advised by Nimble that the array version v5.x will work fine and so left our DR array on v5.x with a pending upgrade that had a blocker due to an issue. This issue was resolved in December and the pending upgrade completed to match the DR array took place around 30 days ago. </div><div><br></div><div><br></div><div>With regards to the production issue, I assumed that the array API has some changes between v4.x and v5.x and it's causing an issue with Cinder due to the API response. Although I have not been able to find out if or what changes there are that may have occurred after the array upgrade, as the documentation for this is Nimble internal-only. </div><div><br></div><div></div><div><br></div><div><b>So with that - some questions if I may:</b></div><div> When Openstack got the 409 error response from the API (as seen in the log below), why would Openstack then proceed to delete the snapshot on the Openstack side? How could I debug this further? I'm not sure what Openstack Cinder is acting on in terns of the response as yet. Maybe Openstack is not specifically looking for the error code in the response? </div><div><br></div><div>The snapshot that got deleted on the openstack side is a problem. Would this be related to the driver? Could it be possible that the driver did not pass the error response to Cinder? </div></div></blockquote><div><br></div><div>Hi Tony,</div><div><br></div><div>This is exactly what happened, and it appears to be a driver bug introduced in queens by [1]. The code in question [2] logs the error, but fails to propagate the exception. As far as the volume manager is concerned, the snapshot deletion was successful.<br></div><div><br></div><div>[1] <a href="https://review.opendev.org/601492" rel="noreferrer" target="_blank">https://review.opendev.org/601492</a></div><div> [2] <a href="https://opendev.org/openstack/cinder/src/branch/stable/queens/cinder/volume/drivers/nimble.py#L1815" rel="noreferrer" target="_blank">https://opendev.org/openstack/cinder/src/branch/stable/queens/cinder/volume/drivers/nimble.py#L1815</a></div><div><br></div><div>Alan<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks in advance. Just for reference, the log snippet is below. </div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="background-color:rgb(0,255,0)">==> volume.log <==</span><br>2020-01-17 16:53:23.718 24723 WARNING py.warnings [req-60fe4335-af66-4c46-9bbd-2408bf4d6f07 df7548ecad684f26b8bc802ba63a9814 87e34c89e6fb41d2af25085b64011a55 - default default] /usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: <a href="https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings" rel="noreferrer" target="_blank">https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings</a><br> InsecureRequestWarning)<br>: <span style="background-color:rgb(255,255,0)">NimbleAPIException: Failed to execute api snapshots/0464a5fd65d75fcfe1000000000000011100001a41: Error Code: 409 Message: Snapshot snapshot-3806efc5-65ca-495a-a87a-baaddc9607d9 for volume volume-5b02db35-8d5c-4ef6-b0e7-2f9b62cac57e has a clone.</span><br>==> api.log <==<br>2020-01-17 16:53:23.769 25242 INFO cinder.api.openstack.wsgi [req-bfcbff34-134b-497e-82b1-082d48f8767f df7548ecad684f26b8bc802ba63a9814 87e34c89e6fb41d2af25085b64011a55 - default default] <a href="http://192.168.53.45:8776/v3/87e34c89e6fb41d2af25085b64011a55/volumes/detail" rel="noreferrer" target="_blank">http://192.168.53.45:8776/v3/87e34c89e6fb41d2af25085b64011a55/volumes/detail</a> returned with HTTP 200<br>2020-01-17 16:53:23.770 25242 INFO eventlet.wsgi.server [req-bfcbff34-134b-497e-82b1-082d48f8767f df7548ecad684f26b8bc802ba63a9814 87e34c89e6fb41d2af25085b64011a55 - default default] 192.168.53.45 "GET /v3/87e34c89e6fb41d2af25085b64011a55/volumes/detail HTTP/1.1" status: 200 len: 4657 time: 0.1152730<br>==> volume.log <==<br>2020-01-17 16:53:23.811 24723 WARNING py.warnings [req-60fe4335-af66-4c46-9bbd-2408bf4d6f07 df7548ecad684f26b8bc802ba63a9814 87e34c89e6fb41d2af25085b64011a55 - default default] /usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: <a href="https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings" rel="noreferrer" target="_blank">https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings</a><br> InsecureRequestWarning)<br>: NimbleAPIException: Failed to execute api snapshots/0464a5fd65d75fcfe1000000000000011100001a41: Error Code: 409 Message: Snapshot snapshot-3806efc5-65ca-495a-a87a-baaddc9607d9 for volume volume-5b02db35-8d5c-4ef6-b0e7-2f9b62cac57e has a clone.<br><span style="background-color:rgb(255,255,0)">2020-01-17 16:53:23.902 24723 ERROR cinder.volume.drivers.nimble [req-60fe4335-af66-4c46-9bbd-2408bf4d6f07 df7548ecad684f26b8bc802ba63a9814 87e34c89e6fb41d2af25085b64011a55 - default default] Re-throwing Exception Failed to execute api snapshots/0464a5fd65d75fcfe1000000000000011100001a41: Error Code: 409 Message: Snapshot snapshot-3806efc5-65ca-495a-a87a-baaddc9607d9 for volume volume-5b02db35-8d5c-4ef6-b0e7-2f9b62cac57e has a clone.: NimbleAPIException: Failed to execute api snapshots/0464a5fd65d75fcfe1000000000000011100001a41: Error Code: 409 Message: Snapshot snapshot-3806efc5-65ca-495a-a87a-baaddc9607d9 for volume volume-5b02db35-8d5c-4ef6-b0e7-2f9b62cac57e has a clone.</span><br>2020-01-17 16:53:23.903 24723 WARNING cinder.volume.drivers.nimble [req-60fe4335-af66-4c46-9bbd-2408bf4d6f07 df7548ecad684f26b8bc802ba63a9814 87e34c89e6fb41d2af25085b64011a55 - default default] Snapshot snapshot-3806efc5-65ca-495a-a87a-baaddc9607d9 : has a clone: NimbleAPIException: Failed to execute api snapshots/0464a5fd65d75fcfe1000000000000011100001a41: Error Code: 409 Message: Snapshot snapshot-3806efc5-65ca-495a-a87a-baaddc9607d9 for volume volume-5b02db35-8d5c-4ef6-b0e7-2f9b62cac57e has a clone.<br>2020-01-17 16:53:23.964 24723 WARNING cinder.quota [req-60fe4335-af66-4c46-9bbd-2408bf4d6f07 df7548ecad684f26b8bc802ba63a9814 87e34c89e6fb41d2af25085b64011a55 - default default] Deprecated: Default quota for resource: snapshots_Nimble-DR is set by the default quota flag: quota_snapshots_Nimble-DR, it is now deprecated. Please use the default quota class for default quota.<br>2020-01-17 16:53:24.054 24723 INFO cinder.volume.manager [req-60fe4335-af66-4c46-9bbd-2408bf4d6f07 df7548ecad684f26b8bc802ba63a9814 87e34c89e6fb41d2af25085b64011a55 - default default] <span style="background-color:rgb(255,255,0)">Delete snapshot completed successfully.</span></blockquote><div><br></div><div><br></div>Regards,<div><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><p style="background-image:initial;background-position:initial;background-repeat:initial"><a name="m_6330074280495010434_m_3038981133509348776_m_-1416124594182612983_m_-1783822804648842471_m_6574831871010562757_m_-5530621010971127278_SignatureSanitizer_SafeHtmlFilter__MailAutoSig" rel="noreferrer"><b><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Tony Pearce</span></b></a><span style="font-size:9pt;font-family:Arial,sans-serif;color:black">
| </span><b><span style="font-family:Arial,sans-serif;color:rgb(0,112,192)"><span style="font-size:9pt">Senior Network Engineer /
Infrastructure Lead</span><span style="font-size:12px"><br></span></span></b><b><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(0,112,192)"><a href="https://www.cinglevue.com" rel="noreferrer" target="_blank">Cinglevue International</a></span></b></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:Arial,sans-serif" lang="EN-US"></span></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Email: </span><span style="font-family:Arial,sans-serif"><a href="mailto:tony.pearce@cinglevue.com" style="color:rgb(17,85,204);font-size:10pt" rel="noreferrer" target="_blank">tony.pearce@cinglevue.com</a><span style="font-size:12.6667px"><br></span></span><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Web: </span><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(17,85,204)"><a href="http://www.cinglevue.com/" rel="noreferrer" target="_blank">http://www.cinglevue.com</a></span><b><span style="font-size:9pt;font-family:Arial,sans-serif;color:rgb(38,66,120)"> </span></b></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:Arial,sans-serif" lang="EN-US"></span></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><b><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Australia</span></b><span style="font-family:Arial,sans-serif"><font color="#000000"><span style="font-size:10pt"> </span></font><span style="font-size:12.6667px"><br></span></span><span style="color:black;font-family:Arial,sans-serif;font-size:10pt">1 Walsh Loop, Joondalup, WA 6027 Australia.</span></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:9.5pt;font-family:Arial,sans-serif" lang="EN-US"></span></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Direct: </span><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(17,85,204)">+61 8 6202 0036</span><span style="font-size:10pt;font-family:Arial,sans-serif;color:black"> | </span><span style="font-size:9.5pt;font-family:Arial,sans-serif" lang="EN-US"></span><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Main: </span><span style="font-size:10pt;font-family:Arial,sans-serif;color:rgb(17,85,204)">+61 8 6202 0024</span></p><p style="background-image:initial;background-position:initial;background-repeat:initial"><span style="font-size:10pt;font-family:Arial,sans-serif;color:black">Note: This email and all attachments are the sole
property of Cinglevue International Pty Ltd. (or any of its subsidiary
entities), and the information contained herein must be considered
confidential, unless specified otherwise. If you are not the intended
recipient, you must not use or forward the information contained in these
documents. If you have received this message in error, please
delete the email and notify the sender.</span><span style="font-size:9.5pt;font-family:Arial,sans-serif" lang="EN-US"></span></p><p>
</p><p> </p></div></div></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div>