<div dir="ltr">Hello Rajat and Sofia,<br><br>Sorry for the late response. <br><br>Thanks to both of you. <br><br>The logs I shared before are from nova only; we do not see anything apart from that in nova logs. Here are the full logs for your ready reference.<br><br>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.189 23213 ERROR nova.volume.cinder [req-59de985e-9db2-4eb5-a82b-31c64f9758a9 14fd679363724aaf9e9f0503820c0b1e a95811c12f3449cf8730331fd3be1c69 - default default] Update attachment failed for attachment cab57e8a-9585-4ca5-9e16-2a75bbf16994. Error: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d) Code: 500: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [req-59de985e-9db2-4eb5-a82b-31c64f9758a9 14fd679363724aaf9e9f0503820c0b1e a95811c12f3449cf8730331fd3be1c69 - default default] [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] Instance failed block device setup: cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] Traceback (most recent call last):</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/nova/compute/manager.py", line 1976, in _prep_block_device</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>driver_block_device.attach_block_devices(</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 874, in attach_block_devices</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>_log_and_attach(device)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 871, in _log_and_attach</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>bdm.attach(*attach_args, **attach_kwargs)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 46, in wrapped</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>ret_val = method(obj, context, *args, **kwargs)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 672, in attach</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>self._do_attach(context, instance, volume, volume_api,</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 657, in _do_attach</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>self._volume_attach(context, volume, connector, instance,</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/nova/virt/block_device.py", line 571, in _volume_attach</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>connection_info = volume_api.attachment_update(</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 396, in wrapper</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>res = method(self, ctx, *args, **kwargs)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 447, in wrapper</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>res = method(self, ctx, attachment_id, *args, **kwargs)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 875, in attachment_update</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>LOG.error('Update attachment failed for attachment '</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 227, in __exit__</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>self.force_reraise()</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/oslo_utils/excutils.py", line 200, in force_reraise</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>raise self.value</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/nova/volume/cinder.py", line 867, in attachment_update</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>attachment_ref = cinderclient(</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/api_versions.py", line 423, in substitution</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>return method.func(obj, *args, **kwargs)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/v3/attachments.py", line 75, in update</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>resp = self._update('/attachments/%s' % id, body)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/base.py", line 312, in _update</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>resp, body = self.api.client.put(url, body=body, **kwargs)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 220, in put</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>return self._cs_request(url, 'PUT', **kwargs)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 205, in _cs_request</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>return self.request(url, method, **kwargs)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>File "/var/lib/openstack/lib/python3.8/site-packages/cinderclient/client.py", line 191, in request</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] <span class="gmail-Apple-converted-space"> </span>raise exceptions.from_response(resp, body)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340] cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume 7c0aed26-3ad0-4a0e-8733-4fcda12d97e1). (HTTP 500) (Request-ID: req-f6eca4ff-3275-4255-bb92-c5bdf49adb8d)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">2022-12-07 10:52:16.214 23213 ERROR nova.compute.manager [instance: 2f0e87a1-c8a3-48ca-b273-9a2d4206c340]<span class="gmail-Apple-converted-space"> </span></span></p></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, 7 Dec 2022 at 16:48, Rajat Dhasmana <<a href="mailto:rdhasman@redhat.com" target="_blank">rdhasman@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Hemant,<div><br></div><div>Sorry for getting back to this so late.</div><div>The problem you've described for instance actions like delete, resize or shelve not deleting the cinder attachment entries.</div><div>I'm not sure about resize and shelve but deleting an instance should delete the volume attachment entry.</div><div>I suggest you check nova logs for the calls it makes to cinder for attachment delete operation and see if there are any errors related to it.</div><div>Ideally it should work but due to some issue, there might be failures.</div><div>If you find a failure in nova logs related to attachment calls, you can go ahead to check cinder logs (api and volume) to see possible</div><div>issues why the attachment wasn't getting deleted on the cinder side.</div><div><br></div><div>-</div><div>Rajat Dhasmana</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Nov 29, 2022 at 2:07 PM Hemant Sonawane <<a href="mailto:hemant.sonawane@itera.io" target="_blank">hemant.sonawane@itera.io</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hello Sofia, <br><br>Thank you for taking it into consideration. Do let me know if you have any questions and updates on the same. <br><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 28 Nov 2022 at 18:12, Sofia Enriquez <<a href="mailto:senrique@redhat.com" target="_blank">senrique@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Hemant,</div><div><br></div><div>Thanks for reporting this issue on the bug tracker <a href="https://bugs.launchpad.net/cinder/+bug/1998083" target="_blank">https://bugs.launchpad.net/cinder/+bug/1998083</a> <br></div><div><br></div><div>I did a quick search and no problems with <font face="arial, sans-serif">shelving operations</font> have been reported for at least the last two years.I'll bring this bug to the cinder bug meeting this week. <br></div><div><br></div><div>Thanks</div><div>Sofia<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 25, 2022 at 1:15 PM Hemant Sonawane <<a href="mailto:hemant.sonawane@itera.io" target="_blank">hemant.sonawane@itera.io</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Rajat,<br><font face="arial, sans-serif">It's not about deleting attachments entries but the normal operations from horizon or via cli does not work because of that. So it really needs to be fixed to perform resize, shelve unshelve operations. <br></font><br>Here are the detailed attachment entries you can see for the shelved instance.<br><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><br></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">+--------------------------------------+--------------------------------------+--------------------------+----------------------------------<b>----+---------------+-----------------------------------------</b></span><b><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures">--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures">| id <span> </span>| volume_id<span> </span>| attached_host<span> </span>| instance_uuid<span> </span>| attach_status | connector <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures">+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures">--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures">| 8daddacc-8fc8-4d2b-a738-d05deb20049f | 67ea3a39-78b8-4d04-a280-166acdc90b8a | <a href="http://nfv1compute43.nfv1.o2.cz" target="_blank">nfv1compute43.nfv1.o2.cz</a> | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | attached<span> </span>| {"platform": "x86_64", "os_type": "linux</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures">", "ip": "10.42.168.87", "host": "<a href="http://nfv1compute43.nfv1.o2.cz" target="_blank">nfv1compute43.nfv1.o2.cz</a>", "multipath": false, "do_local_attach": false, "system uuid": "65917e4f-c8c4-a2af-ec11-fe353e13f4dd", "mountpoint": "/dev/vda"} | <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures">| d3278543-4920-42b7-b217-0858e986fcce | 67ea3a39-78b8-4d04-a280-166acdc90b8a | NULL <span> </span>| 9266a2d7-9721-4994-a6b5-6b3290862dc6 | reserved<span> </span>| NULL<span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures"><span> </span>| <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures">+--------------------------------------+--------------------------------------+--------------------------+--------------------------------------+---------------+-----------------------------------------</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures">--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p>
<p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><b>2 rows in set (0.00 sec) </b><span><b> </b><br><br><br>for e.g if I would like to unshelve this instance it wont work as it has a duplicate entry in cinder db for the attachment. So i have to delete it manually from db or via cli <br><br>
</span></span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures">root@master01:/home/hemant# cinder --os-volume-api-version 3.27 attachment-list --all | grep 67ea3a39-78b8-4d04-a280-166acdc90b8a <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><b><span style="font-variant-ligatures:no-common-ligatures">| 8daddacc-8fc8-4d2b-a738-d05deb20049f | </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)">67ea3a39-78b8-4d04-a280-166acdc90b8a</span><span style="font-variant-ligatures:no-common-ligatures"> | attached | 9266a2d7-9721-4994-a6b5-6b3290862dc6 | <span> </span></span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(203,74,5);background-color:rgb(203,74,5)">│</span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(161,113,161)">·</span></b></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:11px;line-height:normal;font-family:"MesloLGS NF";color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures"><span><b>
</b></span></span></p><p style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;line-height:normal;color:rgb(0,0,0)"><b style="font-size:11px;font-family:"MesloLGS NF""><span style="font-variant-ligatures:no-common-ligatures">| d3278543-4920-42b7-b217-0858e986fcce | </span><span style="font-variant-ligatures:no-common-ligatures;color:rgb(202,51,35)">67ea3a39-78b8-4d04-a280-166acdc90b8a</span></b><span style="font-variant-ligatures:no-common-ligatures"><b style="font-size:11px;font-family:"MesloLGS NF""> | reserved | 9266a2d7-9721-4994-a6b5-6b3290862dc6 |</b><br><br><b style="font-size:11px;font-family:"MesloLGS NF"">cinder --os-volume-api-version 3.27 attachment-delete 8daddacc-8fc8-4d2b-a738-d05deb20049f</b><br><br><font face="arial, sans-serif">this is the only choice I have if I would like to unshelve vm. But this is not a good approach for production envs. I hope you understand me. Please feel free to ask me anything if you don't understand.</font><font style="font-size:11px" face="MesloLGS NF"> </font><br><br><br></span></p></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 25 Nov 2022 at 13:20, Rajat Dhasmana <<a href="mailto:rdhasman@redhat.com" target="_blank">rdhasman@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Hemant,<div><br></div><div>If your final goal is to delete the attachment entries in the cinder DB, we have attachment APIs to perform these tasks. The command useful for you is attachment list[1] and attachment delete[2].</div><div>Make sure you pass the right microversion i.e. 3.27 to be able to execute these operations.</div><div><br></div><div>Eg:</div><div>cinder --os-volume-api-version 3.27 attachment-list</div><div><br></div><div>[1] <a href="https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinder-attachment-list" target="_blank">https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinder-attachment-list</a></div><div>[2] <a href="https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinder-attachment-delete" target="_blank">https://docs.openstack.org/python-cinderclient/latest/cli/details.html#cinder-attachment-delete</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Nov 25, 2022 at 5:44 PM Hemant Sonawane <<a href="mailto:hemant.sonawane@itera.io" target="_blank">hemant.sonawane@itera.io</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello <br>I am using wallaby release openstack and having issues with cinder volumes as once I try to delete, resize or unshelve the shelved vms the volume_attachement entries do not get deleted in cinder db and therefore the above mentioned operations fail every time. I have to delete these volume_attachement entries manually then it works. Is there any way to fix this issue ?<br clear="all"><div><br></div><div>nova-compute logs:<br><br>cinderclient.exceptions.ClientException: Unable to update attachment.(Invalid volume: duplicate connectors detected on volume<br><br>Help will be really appreciated Thanks !</div>-- <br><div dir="ltr"><div dir="ltr">Thanks and Regards,<div><br></div><div>Hemant Sonawane</div><div><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4x8nIGqikcShn_0zvLqn8N5Q_X7samOLi0X9Ztcfux-Zmk47oem97UgQbLfzWjlxQTZ7P3zKhw" width="96" height="37"><br></div></div></div></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr">Thanks and Regards,<div><br></div><div>Hemant Sonawane</div><div><img src="https://ci3.googleusercontent.com/mail-sig/AIorK4x8nIGqikcShn_0zvLqn8N5Q_X7samOLi0X9Ztcfux-Zmk47oem97UgQbLfzWjlxQTZ7P3zKhw" width="96" height="37"><br></div></div></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div>
<p style="font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:capitalize;font-family:RedHatText,sans-serif">
<font size="1"><span>Sofía</span> <span>Enriquez</span><span style="color:rgb(170,170,170);margin:0px"></span>
</font></p><font size="1">
</font><p style="font-weight:normal;margin:0px 0px 4px;text-transform:capitalize;font-family:RedHatText,sans-serif"><font size="1">she/her</font></p><font size="1">
</font><p style="font-weight:normal;margin:0px;text-transform:capitalize;font-family:RedHatText,sans-serif">
<font size="1"><span>Software Engineer</span></font>
</p><font size="1">
</font><p style="font-weight:normal;margin:0px 0px 4px;font-family:RedHatText,sans-serif">
<font size="1"><a style="color:rgb(0,136,206);margin:0px;text-decoration:none;font-family:RedHatText,sans-serif" href="https://www.redhat.com" target="_blank">Red Hat <span>PnT</span></a></font>
</p><font size="1"><span></span></font><p style="font-weight:normal;margin:0px;font-family:RedHatText,sans-serif"><font size="1"><span><span>IRC: @enriquetaso</span></span></font>
</p><font size="1">
</font><div style="font-weight:normal;font-size:12px;font-family:RedHatText,sans-serif;margin-bottom:8px"><font size="1">
</font><div><font size="1">
<a href="https://twitter.com/redhat" title="twitter" style="background:url("https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/3780bd4ede961ef3cd4108b8c0e80186/web-icon-twitter.png") 0px 50%/13px no-repeat transparent;text-decoration:none;color:rgb(0,0,0);display:inline-block;line-height:20px;padding-left:13px" target="_blank"> <span style="margin-left:2px">@RedHat</span> </a> <a href="https://www.linkedin.com/company/red-hat" title="LinkedIn" style="background:url("https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/8d3507e3c6b6c9ad10e301accf1a4af0/web-icon-linkedin.png") 0px 50%/12px no-repeat transparent;text-decoration:none;color:rgb(0,0,0);display:inline-block;line-height:20px;padding-left:12px;margin:0px 4px 0px 3px;padding-top:1px" target="_blank"> <span style="margin-left:4px">Red Hat</span> </a> </font><a href="https://www.facebook.com/RedHatInc" title="Facebook" style="background:url("https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/220b85e2f100025e94cb1bcd993bd51d/web-icon-facebook.png") 0px 50%/11px no-repeat transparent;text-decoration:none;color:rgb(0,0,0);display:inline-block;line-height:20px;padding-left:13px" target="_blank"><font size="1"> <span>Red Hat</span></font> </a>
</div>
</div>
<div style="margin-top:12px">
<table border="0">
<tbody><tr>
<td width="100px"><a href="https://www.redhat.com" target="_blank"> <img src="https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/f5445ae0c9ddafd5b2f1836854d7416a/Logo-RedHat-Email.png" width="90" height="auto"></a> </td>
</tr>
</tbody></table>
</div>
</div><table border="0"><tbody><tr><td width="100px"><br></td>
</tr></tbody></table>
</div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr">Thanks and Regards,<div><br></div><div>Hemant Sonawane</div><div><img width="96" height="37" src="https://ci3.googleusercontent.com/mail-sig/AIorK4x8nIGqikcShn_0zvLqn8N5Q_X7samOLi0X9Ztcfux-Zmk47oem97UgQbLfzWjlxQTZ7P3zKhw"><br></div></div></div>
</div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr"><div dir="ltr">Thanks and Regards,<div><br></div><div>Hemant Sonawane</div><div><img width="96" height="37" src="https://ci3.googleusercontent.com/mail-sig/AIorK4x8nIGqikcShn_0zvLqn8N5Q_X7samOLi0X9Ztcfux-Zmk47oem97UgQbLfzWjlxQTZ7P3zKhw"><br></div></div></div>