<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Hi team!<br>
</p>
<p>We are currently facing an issue in our out-of-tree driver
nova-dpm [1] with nova and cinder on master, where instance launch
in devstack is failing due to communication/time-out issues in
nova-cinder. We are unable to get to the root cause of the issue
and we need your help on getting some hints/directions to debug
this issue further.</p>
<p>--> From nova-compute service: BuildAbortException: Build of
instance aborted: Unable to update the attachment. (HTTP 500) from
the nova-compute server (detailed logs here [2]). <br>
</p>
<p>--> From cinder-volume service: ERROR
oslo_messaging.rpc.server VolumeAttachmentNotFound: Volume
attachment could not be found with filter: attachment_id =
266ef7e1-4735-40f1-b704-509472f565cb.<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server (detailed logs here [3])</p>
Debugging steps done so far:-<br>
<ul>
<li>Compared the package versions between the current devstack
under test with the *<b>last succeeding job in our CI system</b>*
(to be exact, it was for the patches
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/458514/">https://review.openstack.org/#/c/458514/</a> and
<a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/458820/">https://review.openstack.org/#/c/458820/</a>); However the package
versions for packages such as sqlalchemy, os-brick, oslo* are
exactly the same in both the systems.</li>
<li>We used git bisect to revert nova and cinder projects to
versions equal to or before the date of our last succeeding CI
run; but still we were able to reproduce the same error.</li>
<li>Our guess is that the db "Save" operation during the update of
volume attachment is failing. But we are unable to trace/debug
to that point in the rpc call; Any suggestions on how to debug
this sceario would be really helpful.</li>
<li>We are running devstack master on Ubuntu 16.04.04<br>
</li>
</ul>
<br>
<p>References</p>
<p>[1] <a class="moz-txt-link-freetext" href="https://github.com/openstack/nova-dpm">https://github.com/openstack/nova-dpm</a><br>
</p>
<p><br>
</p>
<p>[2] Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.volume.cinder [None req-751d4586-cd97-4a8f-9423-f2bc4b1f1269
service nova] Update attachment failed for attachment
266ef7e1-4735-40f1-b704-509472f565cb. Error: Unable to update the
attachment. (HTTP 500) (Request-ID:
req-550f6b9c-7f22-4dce-adbe-f6843e0aa3ce) Code: 500:
ClientException: Unable to update the attachment. (HTTP 500)
(Request-ID: req-550f6b9c-7f22-4dce-adbe-f6843e0aa3ce)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [None
req-751d4586-cd97-4a8f-9423-f2bc4b1f1269 service nova] [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] Instance failed block device
setup: ClientException: Unable to update the attachment. (HTTP
500) (Request-ID: req-550f6b9c-7f22-4dce-adbe-f6843e0aa3ce)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] Traceback (most recent call
last):<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/compute/manager.py", line 1577, in
_prep_block_device<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2]
wait_func=self._await_block_device_map_created)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/virt/block_device.py", line 828, in
attach_block_devices<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] _log_and_attach(device)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/virt/block_device.py", line 825, in
_log_and_attach<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] bdm.attach(*attach_args,
**attach_kwargs)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/virt/block_device.py", line 46, in wrapped<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] ret_val = method(obj,
context, *args, **kwargs)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/virt/block_device.py", line 618, in attach<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] virt_driver,
do_driver_attach)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/usr/local/lib/python2.7/dist-packages/oslo_concurrency/lockutils.py",
line 274, in inner<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] return f(*args,
**kwargs)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/virt/block_device.py", line 615, in
_do_locked_attach<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] self._do_attach(*args,
**_kwargs)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/virt/block_device.py", line 600, in
_do_attach<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] do_driver_attach)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/virt/block_device.py", line 514, in
_volume_attach<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2]
self['mount_device'])['connection_info']<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/volume/cinder.py", line 291, in wrapper<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] res = method(self, ctx,
*args, **kwargs)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/volume/cinder.py", line 327, in wrapper<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] res = method(self, ctx,
attachment_id, *args, **kwargs)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/volume/cinder.py", line 736, in
attachment_update<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] 'code': getattr(ex,
'code', None)})<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py",
line 220, in __exit__<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] self.force_reraise()<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py",
line 196, in force_reraise<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] six.reraise(self.type_,
self.value, self.tb)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/volume/cinder.py", line 726, in
attachment_update<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] attachment_id,
_connector)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/usr/local/lib/python2.7/dist-packages/cinderclient/v3/attachments.py",
line 67, in update<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] resp =
self._update('/attachments/%s' % id, body)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/usr/local/lib/python2.7/dist-packages/cinderclient/base.py",
line 344, in _update<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] resp, body =
self.api.client.put(url, body=body, **kwargs)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/usr/local/lib/python2.7/dist-packages/cinderclient/client.py",
line 206, in put<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] return
self._cs_request(url, 'PUT', **kwargs)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/usr/local/lib/python2.7/dist-packages/cinderclient/client.py",
line 191, in _cs_request<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] return self.request(url,
method, **kwargs)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/usr/local/lib/python2.7/dist-packages/cinderclient/client.py",
line 177, in request<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] raise
exceptions.from_response(resp, body)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] ClientException: Unable to
update the attachment. (HTTP 500) (Request-ID:
req-550f6b9c-7f22-4dce-adbe-f6843e0aa3ce)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] <br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [None
req-751d4586-cd97-4a8f-9423-f2bc4b1f1269 service nova] [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] Build of instance
d761da60-7bb1-415e-b5b9-eaaed124d6d2 aborted: Unable to update the
attachment. (HTTP 500) (Request-ID:
req-550f6b9c-7f22-4dce-adbe-f6843e0aa3ce): BuildAbortException:
Build of instance d761da60-7bb1-415e-b5b9-eaaed124d6d2 aborted:
Unable to update the attachment. (HTTP 500) (Request-ID:
req-550f6b9c-7f22-4dce-adbe-f6843e0aa3ce)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] Traceback (most recent call
last):<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/compute/manager.py", line 1839, in
_do_build_and_run_instance<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] filter_properties,
request_spec)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/compute/manager.py", line 2052, in
_build_and_run_instance<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2]
bdms=block_device_mapping)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py",
line 220, in __exit__<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] self.force_reraise()<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py",
line 196, in force_reraise<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] six.reraise(self.type_,
self.value, self.tb)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/compute/manager.py", line 2004, in
_build_and_run_instance<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] block_device_mapping) as
resources:<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/usr/lib/python2.7/contextlib.py", line 17, in __enter__<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] return self.gen.next()<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] File
"/opt/stack/nova/nova/compute/manager.py", line 2211, in
_build_resources<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2]
reason=e.format_message())<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] BuildAbortException: Build
of instance d761da60-7bb1-415e-b5b9-eaaed124d6d2 aborted: Unable
to update the attachment. (HTTP 500) (Request-ID:
req-550f6b9c-7f22-4dce-adbe-f6843e0aa3ce)<br>
Apr 25 06:41:57 zos057 nova-compute[6190]: ERROR
nova.compute.manager [instance:
d761da60-7bb1-415e-b5b9-eaaed124d6d2] <br>
</p>
<p><br>
</p>
<p>[3] Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments
[req-f9f3364b-4dd8-4195-a60a-2f0e44c1f2ea
req-550f6b9c-7f22-4dce-adbe-f6843e0aa3ce admin admin] Unable to
update the attachment.: MessagingTimeout: Timed out waiting for a
reply to message ID fe836528e2ea43edabe8004845837f4f<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments Traceback (most recent call last):<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments File
"/opt/stack/cinder/cinder/api/v3/attachments.py", line 228, in
update<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments connector))<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments File
"/opt/stack/cinder/cinder/volume/api.py", line 2158, in
attachment_update<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments attachment_ref.id))<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments File
"/opt/stack/cinder/cinder/rpc.py", line 187, in _wrapper<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments return f(self, *args, **kwargs)<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments File
"/opt/stack/cinder/cinder/volume/rpcapi.py", line 442, in
attachment_update<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments attachment_id=attachment_id)<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py",
line 174, in call<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments retry=self.retry)<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py",
line 131, in _send<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments timeout=timeout, retry=retry)<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 559, in send<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments retry=retry)<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 548, in _send<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments result = self._waiter.wait(msg_id,
timeout)<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 440, in wait<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments message = self.waiters.get(msg_id,
timeout=timeout)<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
line 328, in get<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments 'to message ID %s' % msg_id)<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments MessagingTimeout: Timed out waiting for
a reply to message ID fe836528e2ea43edabe8004845837f4f<br>
Apr 25 06:41:57 zos057 <a class="moz-txt-link-abbreviated" href="mailto:devstack@c-api.service">devstack@c-api.service</a>[11490]: ERROR
cinder.api.v3.attachments <br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server
[req-f9f3364b-4dd8-4195-a60a-2f0e44c1f2ea
req-550f6b9c-7f22-4dce-adbe-f6843e0aa3ce admin None] Exception
during message handling: VolumeAttachmentNotFound: Volume
attachment could not be found with filter: attachment_id =
266ef7e1-4735-40f1-b704-509472f565cb.<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server Traceback (most recent call last):<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py",
line 163, in _process_incoming<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server res =
self.dispatcher.dispatch(message)<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 220, in dispatch<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server return self._do_dispatch(endpoint,
method, ctxt, args)<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server File
"/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py",
line 190, in _do_dispatch<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server result = func(ctxt, **new_args)<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server File
"/opt/stack/cinder/cinder/volume/manager.py", line 4378, in
attachment_update<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server connector)<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server File
"/opt/stack/cinder/cinder/volume/manager.py", line 4349, in
_connection_create<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server
self.db.volume_attachment_update(ctxt, attachment.id, values)<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server File
"/opt/stack/cinder/cinder/db/api.py", line 365, in
volume_attachment_update<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server return
IMPL.volume_attachment_update(context, attachment_id, values)<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server File
"/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 182, in
wrapper<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server return f(*args, **kwargs)<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server File
"/opt/stack/cinder/cinder/db/sqlalchemy/api.py", line 2674, in
volume_attachment_update<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server filter='attachment_id = ' +
attachment_id)<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server VolumeAttachmentNotFound: Volume
attachment could not be found with filter: attachment_id =
266ef7e1-4735-40f1-b704-509472f565cb.<br>
Apr 25 06:42:47 zos057 cinder-volume[11475]: ERROR
oslo_messaging.rpc.server <br>
<br>
</p>
<br>
<pre class="moz-signature" cols="72">--
---------------------------------------------------------------------------------------------------
Sreeram Vancheeswaran
System z Firmware - Openstack Development
IBM Systems & Technology Lab, Bangalore, India
Phone: +91 80 40660826 Mob: +91-9341411511
Email : <a class="moz-txt-link-abbreviated" href="mailto:sreeram@linux.vnet.ibm.com">sreeram@linux.vnet.ibm.com</a></pre>
</body>
</html>