<div dir="ltr">Hi John,<div>That indeed helped. But I'm stuck in using the python interface to make it work:</div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><p class="gmail-p1"><span class="gmail-s1">_opts = {</span><span class="gmail-s2">'object_uu_threads'</span><span class="gmail-s1">: </span><span class="gmail-s2">10</span><span class="gmail-s1">, </span><span class="gmail-s2">'segment_size'</span><span class="gmail-s1">: </span><span class="gmail-s2">1048576</span><span class="gmail-s1">}</span></p></div><div><p class="gmail-p1"><span class="gmail-s1"> </span><span class="gmail-s2">with</span><span class="gmail-s1"> SwiftService(options=_opts) </span><span class="gmail-s2">as</span><span class="gmail-s1"> swift:</span></p></div></blockquote><div>







<p class="gmail-p1"><span class="gmail-s1">Seems not working. </span></p><p class="gmail-p1"><span class="gmail-s1">Also my another question is about the relationship with uu_threads and segment, my understanding is that if we upload by segments, we should use more than 1 threads so that parallelism is possible, right?</span></p><p class="gmail-p1"><span class="gmail-s1"><br></span></p><p class="gmail-p1"><span class="gmail-s1">Best,</span></p><p class="gmail-p1"><span class="gmail-s1">Jialin</span></p></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 5, 2017 at 11:32 AM, John Dickinson <span dir="ltr"><<a href="mailto:me@not.mn" target="_blank">me@not.mn</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>




<div>
<div style="font-family:sans-serif"><div style="white-space:normal"><p dir="auto">I'm not familiar with that format, but in general, if you're dealing with large objects, you can get better performance by splitting the data in the client, uploading each separate segment concurrently, and then creating a large object manifest to tie the segments together (allowing future access by either segments or as a logical whole).</p>
<p dir="auto"><a href="https://docs.openstack.org/swift/latest/overview_large_objects.html#module-swift.common.middleware.slo" style="color:#3983c4" target="_blank">https://docs.openstack.org/<wbr>swift/latest/overview_large_<wbr>objects.html#module-swift.<wbr>common.middleware.slo</a></p><span class="HOEnZb"><font color="#888888">
<p dir="auto">--John</p></font></span><div><div class="h5">
<br><br><br><p dir="auto">On 5 Oct 2017, at 11:24, Jialin Liu wrote:</p>
</div></div></div><div><div class="h5">
<blockquote style="border-left:2px solid #777;color:#777;margin:0 0 5px;padding-left:5px"><div id="m_-6899555455132932131017D90A-1887-460A-B71A-FC38993B5AAA"><div dir="ltr">Thank you John, <div><br></div><div>I think I figured out the way. My case is a little bit rare as I'm dealing with the HDF5 file format, I can not think of any way to retrieve the entire in memory file as a single python object and then serialize it. </div><div>What I did is to use the hdf5.get_file_image function to get a memory image of the file, and then use, io.ByteIO(image) to make it a file-like object. </div><div><br></div><div>So far, it works well, but I believe it is not the best way in terms of performance. </div><div><br></div><div>Best,</div><div>Jialin</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 5, 2017 at 8:25 AM, John Dickinson <span dir="ltr"><<a href="mailto:me@not.mn" target="_blank">me@not.mn</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><u></u>




<div>
<div style="font-family:sans-serif"><div style="white-space:normal">
<p dir="auto">If you've got an arbitrary object in Python, you'll need to serialize it to a file-like object. You could keep it in memory and use a <code style="background-color:#f7f7f7;border-radius:3px;margin:0;padding:0 0.4em" bgcolor="#F7F7F7">StringIO</code> type, or you could serialize it to disk and <code style="background-color:#f7f7f7;border-radius:3px;margin:0;padding:0 0.4em" bgcolor="#F7F7F7">open()</code> it like any other file.</p>

<p dir="auto">Ultimately, Swift is storing arbitrary bytes and doesn't care what they are. You, as the Swift client (i.e. API user), need to dump those bytes on the network to send them to Swift. As long as you're transforming your Python object[s] in some regular way that makes sense in your application, it doesn't matter what bytes you send to Swift.</p>

<p dir="auto">--John</p><div><div class="m_-689955545513293213h5">

<p dir="auto">On 5 Oct 2017, at 8:07, Jialin Liu wrote:</p>

</div></div></div>
<div style="white-space:normal"><blockquote style="border-left:2px solid #777;color:#777;margin:0 0 5px;padding-left:5px"><p dir="auto"></p><div><div class="m_-689955545513293213h5">Hi,<br>
It seems to me that openstack swift only supports file upload/download, is<br>
it possible to put a python object to swift store?<br>
The doc says we could use file-like object, e.g., StringIO, but this is<br>
very limited. I'd like to write a numpy array or other python object into<br>
the swift store, can anybody tell me the solution?<br>
<br>
Best,<br>
Jialin<br></div></div>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" style="color:#777" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><p></p>
</blockquote></div>
<div style="white-space:normal">
</div>
</div>
</div>

<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br></blockquote></div><br></div></div></blockquote>
<div style="white-space:normal"><blockquote style="border-left:2px solid #777;color:#777;margin:0 0 5px;padding-left:5px">
</blockquote><blockquote style="border-left:2px solid #777;color:#777;margin:0 0 5px;padding-left:5px"><p dir="auto">______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" style="color:#777" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a></p>
</blockquote></div>
</div></div></div>
</div>

<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div>