Open Stack

We're currently running Openstack Juno and are experiencing errors when
performing large object copies between Hadoop HDFS and our Swift object
store. While not using the Savana service directly, we are relying upon the
Swift file system extension for Hadoop created as part of the Savana
project.

In each case, the large object copy (using Hadoop's distcp) results in
Swift reporting an Error 413 - Request entity too large.

As a test case, I created a 5.5 GB file of random data and tried to upload
the file to Swift using Swift's CLI command. Once again Swift returned
Error 413. If, however, I explicitly set a segment size on the Swift
command line of 1G, then the file uploads correctly.

When using Hadoop's distcp to move data from HDFS to Swift, the job always
exists with Swift reporting Error 413. Explicitly setting the
fs.swift.service.x.partsize does not appear to make any difference.

My understanding is that Swift should automagically split files greater
that 5G into multiple segments grouped under a metafile but this appears to
not be working. This was working under the Havana release (Ubuntu) using
the Swift File System jar file downloaded from the Marantis web site.  All
current testing is based up the Juno release and when performing a distcp
using the openstack-hadoop jar file shipped as part of the latest hadoop
distros.

Has anyone else seen this behavior?

Thanks,
/ross

-- 
Ross Lillie
Application Software & Architecture Group
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20150505/a0b96a9e/attachment.html>