Open Stack

Hello,

Thanks for Your help but it not helps. I checked that for sure on each swift 
node there is a lot of free space. What can confirm that is fact that when I 
try to create image with size about 1.7GB and I have 
swift_store_large_object_size set to 1GB than there is error (always after 
send first chunk to swift (200MB). When I only change 
swift_store_large_object_size to 2GB and restart glance-api than the same 
image is created correctly (it is then in one big object).

---
Best regards
Sławek Kapłoński
slawek at kaplonski.pl

Dnia wtorek, 30 września 2014 22:28:11 Timur Nurlygayanov pisze:
> Hi Slawek,
> 
> we faced the same error and this is issue with Swift.
> We can see 100% disk usage on the Swift node during the file upload and
> looks like Swift can't send info about status of the file loading in time.
> 
> On our environments we found the workaround for this issue:
> 1. Set  swift_store_large_object_size = 200 in glance.conf.
> 2. Add to Swift proxy-server.conf:
> 
> [DEFAULT]
> ...
> node_timeout = 90
> 
> Probably we can set this value as default value for this parameter instead
> of '30'?
> 
> 
> Regards,
> Timur
> 
> 
> On Tue, Sep 30, 2014 at 7:41 PM, Sławek Kapłoński <slawek at kaplonski.pl>
> 
> wrote:
> > Hello,
> > 
> > I can't find that upload from was previous logs but I now try to upload
> > same image once again. In glance there was exactly same error. In swift
> > logs I have:
> > 
> > Sep 30 17:35:10 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
> > 30/Sep/2014/15/35/10 HEAD /v1/AUTH_7ef5a7661ccd4c069e3ad387a6dceebd/glance
> > HTTP/1.0 204
> > Sep 30 17:35:16 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y
> > 30/Sep/2014/15/35/16 PUT /v1/AUTH_7ef5a7661ccd4c069e3ad387a6dcee
> > bd/glance/fa5dfe09-74f5-4287-9852-d2f1991eebc0-00001 HTTP/1.0 201 - -
> > 
> > Best regards
> > Slawek Kaplonski
> > 
> > W dniu 2014-09-30 17:03, Kuo Hugo napisał(a):
> >> Hi ,
> >> 
> >> Could you please post the log of related requests in Swift's log ???
> >> 
> >> Thanks // Hugo
> >> 
> >> 2014-09-30 22:20 GMT+08:00 Sławek Kapłoński <slawek at kaplonski.pl>:
> >>  Hello,
> >>  
> >>> I'm using openstack havana release and glance with swift backend.
> >>> Today I found that I have problem when I create image with url in
> >>> "--copy-from" when image is bigger than my
> >>> "swift_store_large_object_size" because then glance is trying to
> >>> split image to chunks with size given in
> >>> "swift_store_large_object_chunk_size" and when try to upload first
> >>> chunk to swift I have error:
> >>> 
> >>> 2014-09-30 15:05:29.361 18023 ERROR glance.store.swift [-] Error
> >>> during chunked upload to backend, deleting stale chunks
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift Traceback
> >>> (most recent call last):
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift   File
> >>> "/usr/lib/python2.7/dist-packages/glance/store/swift.py", line 384,
> >>> in add
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> >>> 
> >>>  content_length=content_length)
> >>> 
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift   File
> >>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1234,
> >>> in put_object
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> >>> 
> >>>  response_dict=response_dict)
> >>> 
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift   File
> >>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1143,
> >>> in _retry
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> >>> 
> >>>  reset_func(func, *args, **kwargs)
> >>> 
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift   File
> >>> "/usr/lib/python2.7/dist-packages/swiftclient/client.py", line 1215,
> >>> in _default_reset
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift     %
> >>> (container, obj))
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> >>> ClientException: put_object('glance',
> >>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-00001', ...) failure and no
> >>> ability to reset contents for reupload.
> >>> 2014-09-30 15:05:29.361 18023 TRACE glance.store.swift
> >>> 2014-09-30 15:05:29.362 18023 ERROR glance.store.swift [-] Failed
> >>> to add object to Swift.
> >>> Got error from Swift: put_object('glance',
> >>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-00001', ...) failure and no
> >>> ability to reset contents for reupload.
> >>> 2014-09-30 15:05:29.362 18023 ERROR glance.api.v1.upload_utils [-]
> >>> Failed to upload image 9f56ccec-deeb-4020-95ba-ca7bf1170056
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> >>> Traceback (most recent call last):
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> >>> 
> >>>  File
> >>> 
> >>> "/usr/lib/python2.7/dist-packages/glance/api/v1/upload_utils.py",
> >>> line 101, in upload_data_to_store
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> >>> 
> >>>  store)
> >>> 
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> >>> 
> >>>  File "/usr/lib/python2.7/dist-packages/glance/store/__init__.py",
> >>> 
> >>> line 333, in store_add_to_backend
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> >>> 
> >>>  (location, size, checksum, metadata) = store.add(image_id, data,
> >>> 
> >>> size)
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> >>> 
> >>>  File "/usr/lib/python2.7/dist-packages/glance/store/swift.py",
> >>> 
> >>> line 447, in add
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> >>> 
> >>>  raise glance.store.BackendException(msg)
> >>> 
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils
> >>> BackendException: Failed to add object to Swift.
> >>> 2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils Got
> >>> error from Swift: put_object('glance',
> >>> '9f56ccec-deeb-4020-95ba-ca7bf1170056-00001', ...) failure and no
> >>> ability to reset contents for reupload.
> >>> 
> >>> Does someone of You got same error and know what is solution of it?
> >>> I was searching about that in google but I not found anything what
> >>> could solve my problem.
> >>> 
> >>> --
> >>> Best regards
> >>> Sławek Kapłoński
> >>> slawek at kaplonski.pl
> >>> 
> >>> _______________________________________________
> >>> Mailing list:
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1]
> >>> Post to     : openstack at lists.openstack.org
> >>> Unsubscribe :
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [1]
> >> 
> >> Links:
> >> ------
> >> [1] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> > 
> > --
> > Pozdrawiam
> > Sławek Kapłoński
> > slawek at kaplonski.pl
> > 
> > _______________________________________________
> > Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/
> > openstack
> > Post to     : openstack at lists.openstack.org
> > Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/
> > openstack
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20141001/3cac99b7/attachment.pgp>