<html><body><p>Thanks for the great explanation Timur. I have bookmarked both and will look into it shortly.<br><br><img width="16" height="16" src="cid:1__=8FBBF525DFF206D48f9e8a93df938690918c8FB@" border="0" alt="Inactive hide details for Timur Sufiev ---05/17/2016 10:38:12 AM---Since 10 lines of code > 1000 words, here are references to "><font color="#424282">Timur Sufiev ---05/17/2016 10:38:12 AM---Since 10 lines of code > 1000 words, here are references to 2 patch chains: * New Swift UI file uplo</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Timur Sufiev <tsufiev@mirantis.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">"OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">05/17/2016 10:38 AM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [openstack-dev] [horizon] [javascript] Async file uploads in a new JS-centric UI</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font size="4">Since 10 lines of code > 1000 words, here are references to 2 patch chains:</font><br><br><font size="4">* New Swift UI file upload </font><a href="https://review.openstack.org/#/c/316143/"><u><font size="4" color="#0000FF">https://review.openstack.org/#/c/316143/</font></u></a><br><font size="4">* New Angular Create Image file upload </font><a href="https://review.openstack.org/#/c/317456/"><u><font size="4" color="#0000FF">https://review.openstack.org/#/c/317456/</font></u></a><br><br><font size="4">I like Create Image solution more because it doesn't use Django csrf_exempt and single FileField form hacks just to accept a binary stream on Django side. So CORS offers us a shorter and more elegant solution to the task of file uploads.</font><br><br><font size="4">I got an off-ML feedback that the question / intention of original mail is not clear. My intention / proposal is to adopt the approach used for uploading files in Create Image workflow as the standard way for other interactions (which include file uploads) between Web Clients and OpenStack services.</font><br><br><font size="4">On Sat, May 14, 2016 at 2:48 AM Timur Sufiev <</font><a href="mailto:tsufiev@mirantis.com"><u><font size="4" color="#0000FF">tsufiev@mirantis.com</font></u></a><font size="4">> wrote:</font><ul><font size="4">Hello, fellow web developers!</font><br><br><font size="4">I'd like to share some of my thoughts and findings that I made during playing with ng-file-upload [1] for file uploads in Swift UI and Create Image wizard in Horizon (both Angular). Sorry for a long letter, couldn't put it shorter (TL;DR => go to CONCLUSION section).</font><br><br><font size="4">As a foreword: it's a really useful library, both for customizing stubborn <input type="file"> widget appearance (hello, themability!) and behavior and for the actual file transfer. More on the file transfer below, since it's the topic that really interests me.</font><br><br><font size="4">MAIN PART</font><br><br><font size="4">First, in modern browsers (by modern I mean HTML5 support and particularly FileReader API) you don't need a single-purposed library to upload file asynchronously, both jQuery $.ajax() and vanilla Angular $http calls support it - just pass File()/Blob() object as data (no wrapping {} please) and it works - browser transparently reads data chunk by chunk from a local file system and sends it to the server. There is even a working solution for Django and jQuery-based 'Create Image' form [2]. There are a few subtleties though. Most importantly, there should be no other data (like other key-value pairs from form fields), just the file blob - and then the server endpoint must support raw byte stream as well. This rules out Django views which expect certain headers and body structure.</font><br><br><font size="4">(Second,) What ng-file-upload offers us to solve the challenge of file transfers? There are 2 methods in Upload service: .http() and .upload(). First one is a very thin wrapper around Angular $http, with one difference that it allows to notify() browser of file upload progress (when just a single file blob is passed in .http(), as in case of $http() as well). The second method offers more features, like abortable/resumable uploads and transparent handling of data like {key1: value1, key2: value2, file: FileBlob}. Uploading such data is implemented using standard multipart/form-data content type, so actually, it's just a convenient wrapper around facilities we've already seen. Anyways it's better to just feed the data into Upload.upload() than to construct FormData() on your own (still the same is happening under the bonnet).</font><br><br><font size="4">Third, and most important point, we still have to couple Upload.http() / Upload.upload() with a server-side machinery. If it's a single file upload with Upload.http(), then the server must be able to work with raw binary stream (I'm repeating myself here). If it's a form data including file blob, it's easily handled on front-end with Upload.upload(), then the server must be able to parse multipart/form-data (Django perfectly does that). What's bad in this situation is that it also needs to store any sufficiently sized file in a web server's file system - which is both bug-prone [4] and suboptimal from performance POV. First we need to send a file (possibly GB-sized) from browser to web server, then from web server to the Glance/Swift/any other service host. So, blindly using Upload.upload() won't solve our _real_ problems with file uploads.</font><br><br><font size="4">CONCLUSION</font><br><br><font size="4">What can be done here to help JS UI to handle really large uploads? Split any API calls / views / whatever server things we have into 2 parts: lightweight JSON metadata + heavyweight binary stream. Moreover, use CORS for the second part to send binary streams directly to that data consumers (I know of 2 services atm - Glance & Swift, maybe there are more?). That will reduce the processing time, increasing the possibility that an operation will complete successfully before Keystone token expires :). IMO any new Angular wizard in Horizon should be designed with this thing in mind: a separate API call for binary stream transfer.</font><br><br><font size="4">Thoughts, suggestions?</font><br><br><font size="4">P.S. None of above means that we shouldn't use ng-file-upload, it's still a very convenient tool.</font><br><br><font size="4">[1] </font><a href="https://github.com/danialfarid/ng-file-upload" target="_blank"><u><font size="4" color="#0000FF">https://github.com/danialfarid/ng-file-upload</font></u></a><br><font size="4">[2] </font><a href="https://review.openstack.org/#/c/230434/" target="_blank"><u><font size="4" color="#0000FF">https://review.openstack.org/#/c/230434/</font></u></a><br><font size="4">[3] </font><a href="https://github.com/openstack/horizon/blob/master/horizon/static/horizon/js/horizon.modals.js#L216" target="_blank"><u><font size="4" color="#0000FF">https://github.com/openstack/horizon/blob/master/horizon/static/horizon/js/horizon.modals.js#L216</font></u></a><br><font size="4">[4] </font><a href="https://bugs.launchpad.net/horizon/+bug/1403129" target="_blank"><u><font size="4" color="#0000FF">https://bugs.launchpad.net/horizon/+bug/1403129</font></u></a><br><tt>__________________________________________________________________________<br>OpenStack Development Mailing List (not for usage questions)<br>Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br></tt><tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></tt><tt><br></tt><br><br></ul><BR>
</body></html>