Open Stack

Hi Slawek,

we faced the same error and this is issue with Swift.
We can see 100% disk usage on the Swift node during the file upload and
looks like Swift can't send info about status of the file loading in time.

On our environments we found the workaround for this issue:
1. Set  swift_store_large_object_size = 200 in glance.conf.
2. Add to Swift proxy-server.conf:

[DEFAULT]
...
node_timeout = 90

Probably we can set this value as default value for this parameter instead
of '30'?


Regards,
Timur


On Tue, Sep 30, 2014 at 7:41 PM, Sławek Kapłoński <slawek at kaplonski.pl>
wrote:

> Hello,
>
> I can't find that upload from was previous logs but I now try to upload
> same image once again. In glance there was exactly same error. In swift
> logs I have:
>
> Sep 30 17:35:10 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
> 30/Sep/2014/15/35/10 HEAD /v1/AUTH_7ef5a7661ccd4c069e3ad387a6dceebd/glance
> HTTP/1.0 204
> Sep 30 17:35:16 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
> 30/Sep/2014/15/35/16 PUT /v1/AUTH_7ef5a7661ccd4c069e3ad387a6dcee
> bd/glance/fa5dfe09-74f5-4287-9852-d2f1991eebc0-00001 HTTP/1.0 201 - -
>
> Best regards
> Slawek Kaplonski
>
> W dniu 2014-09-30 17:03, Kuo Hugo napisał(a):
>
>> Hi ,
>>
>> Could you please post the log of related requests in Swift's log ???
>>
>> Thanks // Hugo
>>
>> 2014-09-30 22:20 GMT+08:00 Sławek Kapłoński <slawek at kaplonski.pl>:
>>
>>  Hello,
>>>
>>> I'm using openstack havana release and glance with swift backend.
>>> Today I found that I have problem when I create image with url in
>>> "--copy-from" when image is bigger than my
>>> "swift_store_large_object_size" because then glance is trying to
>>> split image to chunks with size given in
>>> "swift_store_large_object_chunk_size" and when try to upload first
>>> chunk to swift I have error:
>>>
>>> 2014-09-30 15:05:29.361 18023 ERROR glance.store.swift [-] Error
>>> during chunked upload to backend, deleting stale chunks
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift Traceback
>>> (most recent call last):
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift   File
>>> "/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 384,
>>> in add
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
>>>  content_length=content_length)
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift   File
>>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1234,
>>> in put_object
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
>>>  response_dict=response_dict)
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift   File
>>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1143,
>>> in _retry
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
>>>  reset_func(func, *args, **kwargs)
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift   File
>>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1215,
>>> in _default_reset
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift     %
>>> (container, obj))
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
>>> ClientException: put_object('glance',
>>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-00001', ...) failure and no
>>> ability to reset contents for reupload.
>>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
>>> 2014-09-30 15:05:29.362 18023 ERROR glance.store.swift [-] Failed
>>> to add object to Swift.
>>> Got error from Swift: put_object('glance',
>>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-00001', ...) failure and no
>>> ability to reset contents for reupload.
>>> 2014-09-30 15:05:29.362 18023 ERROR glance.api.v1.upload_utils [-]
>>> Failed to upload image 9f56ccec-deeb-4020-95ba-ca7bf1170056
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>> Traceback (most recent call last):
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>>  File
>>> "/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py",
>>> line 101, in upload_data_to_store
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>>  store)
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>>  File "/usr/lib/python2.7/dist-packages/glance/store/__init__.py",
>>> line 333, in store_add_to_backend
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>>  (location, size, checksum, metadata) = store.add(image_id, data,
>>> size)
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>>  File "/usr/lib/python2.7/dist-packages/glance/store/swift.py",
>>> line 447, in add
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>>  raise glance.store.BackendException(msg)
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
>>> BackendException: Failed to add object to Swift.
>>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils Got
>>> error from Swift: put_object('glance',
>>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-00001', ...) failure and no
>>> ability to reset contents for reupload.
>>>
>>> Does someone of You got same error and know what is solution of it?
>>> I was searching about that in google but I not found anything what
>>> could solve my problem.
>>>
>>> --
>>> Best regards
>>> Sławek Kapłoński
>>> slawek at kaplonski.pl
>>>
>>> _______________________________________________
>>> Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1]
>>> Post to     : openstack at lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1]
>>>
>>
>>
>>
>> Links:
>> ------
>> [1] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>
> --
> Pozdrawiam
> Sławek Kapłoński
> slawek at kaplonski.pl
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> openstack
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 

Timur,
QA Engineer
OpenStack Projects
Mirantis Inc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140930/deadac69/attachment.html>