<div dir="ltr"><div><div><div>Hi Slawek,<br><br></div>we faced the same error and this is issue with Swift.<br></div>We can see 100% disk usage on the Swift node during the file upload and looks like Swift can't send info about status of the file loading in time.<br><br></div><div>On our environments we found the workaround for this issue:<br></div><div>1. Set swift_store_large_object_size = 200 in glance.conf.<br><div>2. Add to Swift proxy-server.conf: </div></div><div><p><span class="">[DEFAULT]</span><br>
...<br>
node_timeout = 90<br></p><br></div><div>Probably we can set this value as default value for this parameter instead of '30'?<br><br><br></div><div>Regards,<br></div><div>Timur<br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 30, 2014 at 7:41 PM, Sławek Kapłoński <span dir="ltr"><<a href="mailto:slawek@kaplonski.pl" target="_blank">slawek@kaplonski.pl</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
<br>
I can't find that upload from was previous logs but I now try to upload same image once again. In glance there was exactly same error. In swift logs I have:<br>
<br>
Sep 30 17:35:10 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y 30/Sep/2014/15/35/10 HEAD /v1/AUTH_<u></u>7ef5a7661ccd4c069e3ad387a6dcee<u></u>bd/glance HTTP/1.0 204<br>
Sep 30 17:35:16 127.0.0.1 proxy-server X.X.X.X Y.Y.Y.Y 30/Sep/2014/15/35/16 PUT /v1/AUTH_<u></u>7ef5a7661ccd4c069e3ad387a6dcee<u></u>bd/glance/fa5dfe09-74f5-4287-<u></u>9852-d2f1991eebc0-00001 HTTP/1.0 201 - -<br>
<br>
Best regards<br>
Slawek Kaplonski<br>
<br>
W dniu 2014-09-30 17:03, Kuo Hugo napisał(a):<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Hi ,<br>
<br>
Could you please post the log of related requests in Swift's log ???<br>
<br>
Thanks // Hugo<br>
<br>
2014-09-30 22:20 GMT+08:00 Sławek Kapłoński <<a href="mailto:slawek@kaplonski.pl" target="_blank">slawek@kaplonski.pl</a>>:<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5">
Hello,<br>
<br>
I'm using openstack havana release and glance with swift backend.<br>
Today I found that I have problem when I create image with url in<br>
"--copy-from" when image is bigger than my<br>
"swift_store_large_object_<u></u>size" because then glance is trying to<br>
split image to chunks with size given in<br>
"swift_store_large_object_<u></u>chunk_size" and when try to upload first<br>
chunk to swift I have error:<br>
<br>
2014-09-30 15:05:29.361 18023 ERROR glance.store.swift [-] Error<br>
during chunked upload to backend, deleting stale chunks<br>
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift Traceback<br>
(most recent call last):<br>
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File<br>
"/usr/lib/python2.7/dist-<u></u>packages/glance/store/swift.<u></u>py", line 384,<br>
in add<br>
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift <br>
content_length=content_<u></u>length)<br>
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File<br>
"/usr/lib/python2.7/dist-<u></u>packages/swiftclient/client.<u></u>py", line 1234,<br>
in put_object<br>
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift <br>
response_dict=response_dict)<br>
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File<br>
"/usr/lib/python2.7/dist-<u></u>packages/swiftclient/client.<u></u>py", line 1143,<br>
in _retry<br>
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift <br>
reset_func(func, *args, **kwargs)<br>
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift File<br>
"/usr/lib/python2.7/dist-<u></u>packages/swiftclient/client.<u></u>py", line 1215,<br>
in _default_reset<br>
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift %<br>
(container, obj))<br>
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift<br>
ClientException: put_object('glance',<br>
'9f56ccec-deeb-4020-95ba-<u></u>ca7bf1170056-00001', ...) failure and no<br>
ability to reset contents for reupload.<br>
2014-09-30 15:05:29.361 18023 TRACE glance.store.swift<br>
2014-09-30 15:05:29.362 18023 ERROR glance.store.swift [-] Failed<br>
to add object to Swift.<br>
Got error from Swift: put_object('glance',<br>
'9f56ccec-deeb-4020-95ba-<u></u>ca7bf1170056-00001', ...) failure and no<br>
ability to reset contents for reupload.<br>
2014-09-30 15:05:29.362 18023 ERROR glance.api.v1.upload_utils [-]<br>
Failed to upload image 9f56ccec-deeb-4020-95ba-<u></u>ca7bf1170056<br>
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils<br>
Traceback (most recent call last):<br>
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils <br>
File<br>
"/usr/lib/python2.7/dist-<u></u>packages/glance/api/v1/upload_<u></u>utils.py",<br>
line 101, in upload_data_to_store<br>
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils <br>
store)<br>
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils <br>
File "/usr/lib/python2.7/dist-<u></u>packages/glance/store/__init__<u></u>.py",<br>
line 333, in store_add_to_backend<br>
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils <br>
(location, size, checksum, metadata) = store.add(image_id, data,<br>
size)<br>
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils <br>
File "/usr/lib/python2.7/dist-<u></u>packages/glance/store/swift.<u></u>py",<br>
line 447, in add<br>
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils <br>
raise glance.store.BackendException(<u></u>msg)<br>
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils<br>
BackendException: Failed to add object to Swift.<br>
2014-09-30 15:05:29.362 18023 TRACE glance.api.v1.upload_utils Got<br>
error from Swift: put_object('glance',<br>
'9f56ccec-deeb-4020-95ba-<u></u>ca7bf1170056-00001', ...) failure and no<br>
ability to reset contents for reupload.<br>
<br>
Does someone of You got same error and know what is solution of it?<br>
I was searching about that in google but I not found anything what<br>
could solve my problem.<br>
<br>
--<br>
Best regards<br>
Sławek Kapłoński<br>
<a href="mailto:slawek@kaplonski.pl" target="_blank">slawek@kaplonski.pl</a><br>
<br>
______________________________<u></u>_________________<br>
Mailing list:<br>
</div></div><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a> [1]<br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe :<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a> [1]<br>
</blockquote>
<br>
<br>
<br>
Links:<br>
------<br>
[1] <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><span class="HOEnZb"><font color="#888888"><br>
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
-- <br>
Pozdrawiam</font></span><span class="im HOEnZb"><br>
Sławek Kapłoński<br>
<a href="mailto:slawek@kaplonski.pl" target="_blank">slawek@kaplonski.pl</a><br>
<br>
______________________________<u></u>_________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org" target="_blank">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack</a><br></span><div class="HOEnZb"><div class="h5">
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div dir="ltr"><font color="#888888"><font color="#888888"><br></font></font><div style="font-family:arial;font-size:small">Timur,</div><div style="font-family:arial;font-size:small">QA Engineer</div><div style="font-family:arial;font-size:small">OpenStack Projects</div><div style="font-family:arial;font-size:small">Mirantis Inc</div></div>
</div>