<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Hi Swift Developpers,</div><div class=""><br class=""></div><div class="">We have been using Swift as a IAAS provider for more than two years now, but this mail is about feedback on the API side. I think it would be great to include some of the ideas in future revisions of API.</div><div class=""><br class=""></div><div class="">I’ve been developping a few Swift clients in HTML (in Cloudwatt Dashboard) with CORS, Java with Swing GUI (<a href="https://github.com/pierresouchay/swiftbrowser" class="">https://github.com/pierresouchay/swiftbrowser</a>) and Go for Swift to filesystem (<a href="https://github.com/pierresouchay/swiftsync/" class="">https://github.com/pierresouchay/swiftsync/</a>), so I have now a few ideas about how improving a bit the API.</div><div class=""><br class=""></div><div class="">The API is quite straightforward and intuitive to use, and writing a client is now that difficult, but unfortunately, the Large Object support is not easy at all to deal with.</div><div class=""><br class=""></div><div class="">The biggest issue is that there is now way to know whenever a file is a large object when performing listings using JSON format, since, AFAIK a large object is an object with 0 bytes (so its size in bytes is 0), but it also has a hash of a zero file bytes.</div><div class=""><br class=""></div><div class="">For instance, a signature of such object is :</div><div class=""><font face="Courier New" class=""> {"hash": "d41d8cd98f00b204e9800998ecf8427e", "last_modified": "2015-06-04T10:23:57.618760", "bytes": 0, "name": "5G", "content_type": </font><span style="font-family: 'Courier New';" class="">"</span><span style="font-family: 'Courier New';" class="">octet/stream</span><span style="font-family: 'Courier New';" class="">"</span><span style="font-family: 'Courier New';" class="">}</span></div><div class=""><br class=""></div><div class="">which is, exactly the hash of a 0 bytes file :</div><div class=""><div class=""><font face="Courier New" class="">$ echo -n | md5</font></div><div class=""><font face="Courier New" class="">d41d8cd98f00b204e9800998ecf8427e</font></div></div><div class=""><br class=""></div><div class="">Ok, now lets try HEAD :</div><div class=""><font face="Courier New" class="">$ curl -vv -XHEAD -H X-Auth-Token:$TOKEN '<a href="https://storage.fr1.cloudwatt.com/v1/AUTH_61b8fe6dfd0a4ce69f6622ea74444e0f/large_files/5G" class="">https://storage.fr1.cloudwatt.com/v1/AUTH_61b8fe6dfd0a4ce69f6622ea74444e0f/large_files/5G</a></font></div><div class=""><font face="Courier New" class="">…</font></div><div class=""><div class=""><font face="Courier New" class="">< HTTP/1.1 200 OK</font></div><div class=""><font face="Courier New" class="">< Date: Fri, 09 Oct 2015 19:43:09 GMT</font></div><div class=""><font face="Courier New" class="">< Content-Length: 5000000000</font></div><div class=""><font face="Courier New" class="">< Accept-Ranges: bytes</font></div><div class=""><font face="Courier New" class="">< X-Object-Manifest: large_files/5G/.part-5000000000-</font></div><div class=""><font face="Courier New" class="">< Last-Modified: Thu, 04 Jun 2015 10:16:33 GMT</font></div><div class=""><font face="Courier New" class="">< Etag: "479517ec4767ca08ed0547dca003d116"</font></div><div class=""><font face="Courier New" class="">< X-Timestamp: 1433413437.61876</font></div><div class=""><font face="Courier New" class="">< Content-Type: octet/stream</font></div><div class=""><font face="Courier New" class="">< X-Trans-Id: txba36522b0b7743d683a5d-00561818cd</font></div></div><div class=""><br class=""></div><div class="">WTF ? While all files have the same value for ETag and hash, this is not the case for Large files…</div><div class=""><br class=""></div><div class="">Furthermore, the ETag is not the md5 of the whole file, but the hash of the hash of all manifest files (as described somewhere hidden deeply in the documentation)</div><div class=""><br class=""></div><div class="">Why this is a problem ?</div><div class="">-------------------------------</div><div class=""><br class=""></div><div class="">Imagine a « naive » client using the API which performs some kind of Sync.</div><div class=""><br class=""></div><div class="">The client download each file and when it syncs, compares the local md5 to the md5 of the listing… of course, the hash is the hash of a zero bytes files… so it downloads the file again… and again… and again. Unfortunaly for our naive client, this is exactly the kind of files we don’t want to download twice… since the file is probably huge (after all, it has been split for a reason no ?)</div><div class=""><br class=""></div><div class="">I think this is really a design flaw since you need to know everything about Swift API and extensions to have a proper behavior. The minimum would be to at least return the same value as the ETag header.</div><div class=""><br class=""></div><div class="">OK, let’s continue…</div><div class=""><br class=""></div><div class="">We are not so Naive… our Swift Sync client know that 0 files needs more work.</div><div class=""><br class=""></div><div class="">* First issue: we have to know whenever the file is a « real » 0 bytes file or not. You may think most people do not create 0 bytes files after all… this is dummy. Actually, some I have seen two Object Storage middleware using many 0 bytes files (for instance to store meta data or two set up some kind of directory like structure). So, in this cas, we need to perform a HEAD request to each 0 bytes files. If you have 1000 files like this, you have to perform 1000 HEAD requests to finally know that there are not any Large file. Not very efficient. Your Swift Sync client took 1 second to sync 20G of data with naive approach, now, you need 5 minutes… hash of 0 bytes is not a good idea at all.</div><div class=""><br class=""></div><div class="">* Second issue: since the hash is the hash of all parts (I have an idea about why this decision was made, probably for performance reasons), your client cannot work on files since the hash of local file is not the hash of the Swift aggregated file (which is the hash of all the hash of manifest). So, it means you cannot work on existing data, you have to either :</div><div class=""> - split all the files in the same way as the manifest, compute the MD5 of each part, than compute the MD5 of the hashes and compare to the MD5 on server… (ok… doable, but I gave up with such system)</div><div class=""> - have a local database in your client (when you download, store the REAL Hash of file and store that in fact you have to compare it the the HASH returned by server)</div><div class=""> - perform some kind of crappy heuristics (size + grab the starting bytes of each data of each part or something like that…)</div><div class=""><br class=""></div><div class="">* Third issue:</div><div class=""> - If you don’t want to store the parts of your object file, you have to wait for all your HEAD requests to finish since it is the only way to guess all the files that are referenced in your manifest headers.</div><div class=""><br class=""></div><div class="">So summarize, I think the current API really need some refinements about the listings since a competent developper may trust the bytes value and the hash value and create an algorithm that does not behave nicely. So, the API looks easy but is in fact much more complicated than expected.</div><div class=""><br class=""></div><div class="">A few ideas to improve it :</div><div class=""><br class=""></div><div class="">In listings, if an Object is a large object.</div><div class=""> - either put the real MD5 of file if it is doable technically… or remove it (so naive program will work nicely)… same thing about bytes.</div><div class=""> - add an optional field in the JSON to tell the object is in fact a large object. A nice field to explain the object is a large object would be to use the object-manifest header value. So a client could know the file is a large file or simply a zero byte object, and also know what are the object that are in facts parts of a larger one (and do not wait for you thousands of HEAD requests to finish)</div><div class=""><br class=""></div><div class="">Finally, to help people creating interfaces quickly, add an Option to add CORS for all containers of an account. In our Cloud provider, we added a REST CALL in another WebService with CORS enabled that ensures a container has CORS setup for a Container. So, browsing Swift with HTML5 interfaces is easy. By doing so, it would - I think - greatly increase the Swift Usage (by not needing any specific software to browse Swift).</div><div class=""><br class=""></div><div class="">Best Regards</div><div class=""><br class=""></div><br class=""><div apple-content-edited="true" class="">
<div class="">-- <br class="">Pierre Souchay <<a href="mailto:pierre.souchay@cloudwatt.com" class="">pierre.souchay@cloudwatt.com</a>><br class="">Software Architect @ CloudWatt<br class=""><br class="">Adresse : ETIK 892, Rue Yves Kermen 92100 Boulogne-Billancourt<br class="">N° Standard : +33 1 84 01 04 04<br class="">N° Fax : +33 1 84 01 04 05</div></div></body></html>