[openstack-dev] [python-swiftclient] download multiple files at once (in-memory)

Gerlando FALAUTO gerlando.falauto at st.com
Tue Aug 8 06:48:31 UTC 2017


Hi everyone,

I'm using python-swiftclient and I'd like to be able to download several (small) files.
Actually I don't need/want to have those files saved to disk, I'd rather be able to use their contents in memory for later inline processing through a Jupyter notebook.
I can already download one file at a time by using swiftclient.SwiftService's .download() by passing options  = {"out_file": "-"},
provided my objects list only contains one file.
However, for efficiency reasons (i.e. high latency) I'd like to run several downloads in parallel.

I've identified the offending code block in https://github.com/openstack/python-swiftclient/blob/master/swiftclient/service.py:

            if options['out_file'] and len(objects) > 1:
options['out_file'] = None

I believe this should be something like:
            if options['out_file'] and len(objects) > 1 and options['out_file'] != '-':
options['out_file'] = None

Then everything else should actually work out of the box, I assume.
Does that make any sense at all?

Thank you!
Gerlando
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170808/e51e9f02/attachment.html>


More information about the OpenStack-dev mailing list