Open Stack

Thanks for the great explanation Timur. I have bookmarked both and will
look into it shortly.



From:	Timur Sufiev <tsufiev at mirantis.com>
To:	"OpenStack Development Mailing List (not for usage questions)"
            <openstack-dev at lists.openstack.org>
Date:	05/17/2016 10:38 AM
Subject:	Re: [openstack-dev] [horizon] [javascript] Async file uploads
            in a	new JS-centric UI



Since 10 lines of code > 1000 words, here are references to 2 patch chains:

* New Swift UI file upload https://review.openstack.org/#/c/316143/
* New Angular Create Image file upload
https://review.openstack.org/#/c/317456/

I like Create Image solution more because it doesn't use Django csrf_exempt
and single FileField form hacks just to accept a binary stream on Django
side. So CORS offers us a shorter and more elegant solution to the task of
file uploads.

I got an off-ML feedback that the question / intention of original mail is
not clear. My intention / proposal is to adopt the approach used for
uploading files in Create Image workflow as the standard way for other
interactions (which include file uploads) between Web Clients and OpenStack
services.

On Sat, May 14, 2016 at 2:48 AM Timur Sufiev <tsufiev at mirantis.com> wrote:
  Hello, fellow web developers!

  I'd like to share some of my thoughts and findings that I made during
  playing with ng-file-upload [1] for file uploads in Swift UI and Create
  Image wizard in Horizon (both Angular). Sorry for a long letter, couldn't
  put it shorter (TL;DR => go to CONCLUSION section).

  As a foreword: it's a really useful library, both for customizing
  stubborn <input type="file"> widget appearance (hello, themability!) and
  behavior and for the actual file transfer. More on the file transfer
  below, since it's the topic that really interests me.

  MAIN PART

  First, in modern browsers (by modern I mean HTML5 support and
  particularly FileReader API) you don't need a single-purposed library to
  upload file asynchronously, both jQuery $.ajax() and vanilla Angular
  $http calls support it - just pass File()/Blob() object as data (no
  wrapping {} please) and it works - browser transparently reads data chunk
  by chunk  from a local file system and sends it to the server. There is
  even a working solution for Django and jQuery-based 'Create Image' form
  [2]. There are a few subtleties though. Most importantly, there should be
  no other data (like other key-value pairs from form fields), just the
  file blob - and then the server endpoint must support raw byte stream as
  well. This rules out Django views which expect certain headers and body
  structure.

  (Second,) What ng-file-upload offers us to solve the challenge of file
  transfers? There are 2 methods in Upload service: .http() and .upload().
  First one is a very thin wrapper around Angular $http, with one
  difference that it allows to notify() browser of file upload progress
  (when just a single file blob is passed in .http(), as in case of $http()
  as well). The second method offers more features, like
  abortable/resumable uploads and transparent handling of data like {key1:
  value1, key2: value2, file: FileBlob}. Uploading such data is implemented
  using standard multipart/form-data content type, so actually, it's just a
  convenient wrapper around facilities we've already seen. Anyways it's
  better to just feed the data into Upload.upload() than to construct
  FormData() on your own (still the same is happening under the bonnet).

  Third, and most important point, we still have to couple Upload.http() /
  Upload.upload() with a server-side machinery. If it's a single file
  upload with Upload.http(), then the server must be able to work with raw
  binary stream (I'm repeating myself here). If it's a form data including
  file blob, it's easily handled on front-end with Upload.upload(), then
  the server must be able to parse multipart/form-data (Django perfectly
  does that). What's bad in this situation is that it also needs to store
  any sufficiently sized file in a web server's file system - which is both
  bug-prone [4] and suboptimal from performance POV. First we need to send
  a file (possibly GB-sized) from browser to web server, then from web
  server to the Glance/Swift/any other service host. So, blindly using
  Upload.upload() won't solve our _real_ problems with file uploads.

  CONCLUSION

  What can be done here to help JS UI to handle really large uploads? Split
  any API calls / views / whatever server things we have into 2 parts:
  lightweight JSON metadata + heavyweight binary stream. Moreover, use CORS
  for the second part to send binary streams directly to that data
  consumers (I know of 2 services atm - Glance & Swift, maybe there are
  more?). That will reduce the processing time, increasing the possibility
  that an operation will complete successfully before Keystone token
  expires :). IMO any new Angular wizard in Horizon should be designed with
  this thing in mind: a separate API call for binary stream transfer.

  Thoughts, suggestions?

  P.S. None of above means that we shouldn't use ng-file-upload, it's still
  a very convenient tool.

  [1] https://github.com/danialfarid/ng-file-upload
  [2] https://review.openstack.org/#/c/230434/
  [3]
  https://github.com/openstack/horizon/blob/master/horizon/static/horizon/js/horizon.modals.js#L216
  [4] https://bugs.launchpad.net/horizon/+bug/1403129
  __________________________________________________________________________

  OpenStack Development Mailing List (not for usage questions)
  Unsubscribe:
  OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160517/3e6085a8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160517/3e6085a8/attachment.gif>