Open Stack

Hello, fellow web developers!

I'd like to share some of my thoughts and findings that I made during
playing with ng-file-upload [1] for file uploads in Swift UI and Create
Image wizard in Horizon (both Angular). Sorry for a long letter, couldn't
put it shorter (TL;DR => go to CONCLUSION section).

As a foreword: it's a really useful library, both for customizing stubborn
<input type="file"> widget appearance (hello, themability!) and behavior
and for the actual file transfer. More on the file transfer below, since
it's the topic that really interests me.

MAIN PART

First, in modern browsers (by modern I mean HTML5 support and particularly
FileReader API) you don't need a single-purposed library to upload file
asynchronously, both jQuery $.ajax() and vanilla Angular $http calls
support it - just pass File()/Blob() object as data (no wrapping {} please)
and it works - browser transparently reads data chunk by chunk  from a
local file system and sends it to the server. There is even a working
solution for Django and jQuery-based 'Create Image' form [2]. There are a
few subtleties though. Most importantly, there should be no other data
(like other key-value pairs from form fields), just the file blob - and
then the server endpoint must support raw byte stream as well. This rules
out Django views which expect certain headers and body structure.

(Second,) What ng-file-upload offers us to solve the challenge of file
transfers? There are 2 methods in Upload service: .http() and .upload().
First one is a very thin wrapper around Angular $http, with one difference
that it allows to notify() browser of file upload progress (when just a
single file blob is passed in .http(), as in case of $http() as well). The
second method offers more features, like abortable/resumable uploads and
transparent handling of data like {key1: value1, key2: value2, file:
FileBlob}. Uploading such data is implemented using standard
multipart/form-data content type, so actually, it's just a convenient
wrapper around facilities we've already seen. Anyways it's better to just
feed the data into Upload.upload() than to construct FormData() on your own
(still the same is happening under the bonnet).

Third, and most important point, we still have to couple Upload.http() /
Upload.upload() with a server-side machinery. If it's a single file upload
with Upload.http(), then the server must be able to work with raw binary
stream (I'm repeating myself here). If it's a form data including file
blob, it's easily handled on front-end with Upload.upload(), then the
server must be able to parse multipart/form-data (Django perfectly does
that). What's bad in this situation is that it also needs to store any
sufficiently sized file in a web server's file system - which is both
bug-prone [4] and suboptimal from performance POV. First we need to send a
file (possibly GB-sized) from browser to web server, then from web server
to the Glance/Swift/any other service host. So, blindly using
Upload.upload() won't solve our _real_ problems with file uploads.

CONCLUSION

What can be done here to help JS UI to handle really large uploads? Split
any API calls / views / whatever server things we have into 2 parts:
lightweight JSON metadata + heavyweight binary stream. Moreover, use CORS
for the second part to send binary streams directly to that data consumers
(I know of 2 services atm - Glance & Swift, maybe there are more?). That
will reduce the processing time, increasing the possibility that an
operation will complete successfully before Keystone token expires :). IMO
any new Angular wizard in Horizon should be designed with this thing in
mind: a separate API call for binary stream transfer.

Thoughts, suggestions?

P.S. None of above means that we shouldn't use ng-file-upload, it's still a
very convenient tool.

[1] https://github.com/danialfarid/ng-file-upload
[2] https://review.openstack.org/#/c/230434/
[3]
https://github.com/openstack/horizon/blob/master/horizon/static/horizon/js/horizon.modals.js#L216
[4] https://bugs.launchpad.net/horizon/+bug/1403129
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160513/55dc9eb7/attachment.html>