[openstack-dev] [third-party-ci]Tests randomly failing because of lvm

Andrey Pavlov andrey.mp at gmail.com
Mon Aug 17 07:17:21 UTC 2015


glance log -

2015-08-17 03:13:42.938 2312 INFO swiftclient
[req-463d065c-8c26-4be2-8e30-5f1d83e4a6b8 2c757dc5add24d02b867436d82db5e99
98a460b55a544902b4bfbb104e8fae7f - - -] REQ: curl -i
http://127.0.0.1:8080/v1/AUTH_93bdc098999d400b9838abb51a7a8126/glance/32be6815-3571-48e8-b664-7902613ffd04
-X PUT -H "X-Auth-Token: 5103ad1cf0684071aa47e36683004ead"
2015-08-17 03:13:42.938 2312 INFO swiftclient
[req-463d065c-8c26-4be2-8e30-5f1d83e4a6b8 2c757dc5add24d02b867436d82db5e99
98a460b55a544902b4bfbb104e8fae7f - - -] RESP STATUS: 503 Service Unavailable
2015-08-17 03:13:42.938 2312 INFO swiftclient
[req-463d065c-8c26-4be2-8e30-5f1d83e4a6b8 2c757dc5add24d02b867436d82db5e99
98a460b55a544902b4bfbb104e8fae7f - - -] RESP HEADERS: [('date', 'Mon, 17
Aug 2015 03:13:17 GMT'), ('content-length', '118'), ('content-type',
'text/html; charset=UTF-8'), ('x-trans-id',
'tx78adec8088444befb8faf-0055d15136')]
2015-08-17 03:13:42.938 2312 INFO swiftclient
[req-463d065c-8c26-4be2-8e30-5f1d83e4a6b8 2c757dc5add24d02b867436d82db5e99
98a460b55a544902b4bfbb104e8fae7f - - -] RESP BODY: <html><h1>Service
navailable</h1><p>The server is currently unavailable. Please try again at
a later time.</p></html>
2015-08-17 03:13:43.939 2312 ERROR glance_store._drivers.swift.store
[req-463d065c-8c26-4be2-8e30-5f1d83e4a6b8 2c757dc5add24d02b867436d82db5e99
98a460b55a544902b4bfbb104e8fae7f - - -] Failed to add object to Swift.
Got error from Swift: put_object('glance',
'32be6815-3571-48e8-b664-7902613ffd04', ...) failure and no ability to
reset contents for reupload..
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils
[req-463d065c-8c26-4be2-8e30-5f1d83e4a6b8 2c757dc5add24d02b867436d82db5e99
98a460b55a544902b4bfbb104e8fae7f - - -] Failed to upload image
32be6815-3571-48e8-b664-7902613ffd04
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils Traceback
(most recent call last):
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils   File
"/opt/stack/new/glance/glance/api/v1/upload_utils.py", line 113, in
upload_data_to_store
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils
context=req.context)
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils   File
"/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 340,
in store_add_to_backend
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils
context=context)
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils   File
"/usr/local/lib/python2.7/dist-packages/glance_store/capabilities.py", line
226, in op_checker
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils     return
store_op_fun(store, *args, **kwargs)
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils   File
"/usr/local/lib/python2.7/dist-packages/glance_store/_drivers/swift/store.py",
line 620, in add
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils     raise
glance_store.BackendException(msg)
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils
BackendException: Failed to add object to Swift.
2015-08-17 03:13:43.940 2312 ERROR glance.api.v1.upload_utils Got error
from Swift: put_object('glance', '32be6815-3571-48e8-b664-7902613ffd04',
...) failure and no ability to reset contents for reupload.

swift-object log -

object-server: 127.0.0.1 - - [17/Aug/2015:03:13:12 +0000] "PUT
/sdb1/419/AUTH_93bdc098999d400b9838abb51a7a8126/glance/68b9d0f8-f20f-42e8-a430-8930608e9ed4"
201 - "PUT
http://127.0.0.1:8080/v1/AUTH_93bdc098999d400b9838abb51a7a8126/glance/68b9d0f8-f20f-42e8-a430-8930608e9ed4"
"txe6ba789f3616452aa2a12-0055d15148" "proxy-server 2111" 0.0039 "-" 2159 0
object-server: ERROR __call__ error with PUT
/sdb1/134/AUTH_93bdc098999d400b9838abb51a7a8126/glance/32be6815-3571-48e8-b664-7902613ffd04
: [Errno 28] No space left on device (txn:
tx78adec8088444befb8faf-0055d15136)

maybe test tries to upload very huge image?


On Mon, Aug 17, 2015 at 10:02 AM, Abhishek Shrivastava <
abhishek at cloudbyte.com> wrote:

> Hi Eduard,
>
> This is what I gathered the cause of the test failure:
>
> 2015-08-17 03:13:44.239 ERROR oslo_messaging.rpc.dispatcher [req-95c1bc0f-e333-493b-9730-cfba9c3dfd9a tempest-VolumesV1ActionsTest-418947994] Exception during message handling: 500 Internal Server Error: Failed to upload image 32be6815-3571-48e8-b664-7902613ffd04 (HTTP 500)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/new/cinder/cinder/volume/manager.py", line 976, in copy_volume_to_image
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     payload['message'] = six.text_type(error)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/oslo_utils/excutils.py", line 119, in __exit__
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     six.reraise(self.type_, self.value, self.tb)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/new/cinder/cinder/volume/manager.py", line 963, in copy_volume_to_image
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     image_meta)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 105, in wrapper
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/new/cinder/cinder/volume/drivers/lvm.py", line 449, in copy_volume_to_image
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     self.local_path(volume))
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/new/cinder/cinder/image/image_utils.py", line 351, in upload_volume
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     image_service.update(context, image_id, {}, image_file)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/new/cinder/cinder/image/glance.py", line 334, in update
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     _reraise_translated_image_exception(image_id)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/new/cinder/cinder/image/glance.py", line 516, in _reraise_translated_image_exception
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     six.reraise(type(new_exc), new_exc, exc_trace)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/new/cinder/cinder/image/glance.py", line 332, in update
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     **image_meta)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/opt/stack/new/cinder/cinder/image/glance.py", line 177, in call
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     return getattr(controller, method)(*args, **kwargs)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/glanceclient/v1/images.py", line 360, in update
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     resp, body = self.client.put(url, headers=hdrs, data=image_data)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/glanceclient/common/http.py", line 268, in put
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     return self._request('PUT', url, **kwargs)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher   File "/usr/local/lib/python2.7/dist-packages/glanceclient/common/http.py", line 230, in _request
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher     raise exc.from_response(resp, resp.text)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher HTTPInternalServerError: 500 Internal Server Error: Failed to upload image 32be6815-3571-48e8-b664-7902613ffd04 (HTTP 500)
> 2015-08-17 03:13:44.239 16950 ERROR oslo_messaging.rpc.dispatcher
>
>
> On Mon, Aug 17, 2015 at 12:18 PM, Eduard Matei <
> eduard.matei at cloudfounders.com> wrote:
>
>> Hi,
>>
>> Not sure what snippet you mean but here are all the logs for the failed
>> test run:
>>
>> http://packages.cloudfounders.com/ci_logs/01/213501/3/check/dsvm-tempest-full/37aa6fe/
>> (console.html and c-vol.txt show more details).
>>
>> Thanks,
>> Eduard
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
>
>
> *Thanks & Regards,*
> *Abhishek*
> *Cloudbyte Inc. <http://www.cloudbyte.com>*
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>


-- 
Kind regards,
Andrey Pavlov.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150817/687c1247/attachment.html>


More information about the OpenStack-dev mailing list