
<html>

<head>

<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">

</head>

<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">

All,

<div><br>

</div>

<div>I’m trying to integrate and test access to/from our Swift storage ring via our Hadoop cluster.</div>

<div><br>

</div>

<div>After setting the HDFS permissions, I’m able to perform the following tests using the Swift/Hadoop file system:</div>

<div><br>

</div>

<div><span class="Apple-tab-span" style="white-space:pre"></span>distcp swift://{container}.{provider}/object swift://{container}.{provider}/new-object</div>

<div><span class="Apple-tab-span" style="white-space:pre"></span>distcp hdfs://{node}:{port}/object swift://{container}.{provider}/new-object</div>

<div><br>

</div>

<div>However, I’m not able to use distcp to move objects from Swift to HDFS storage. The log file from the final attempt to create the associated map task is as follows:</div>

<div><br>

</div>

2014-02-25 22:10:38,570 DEBUG org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem: Initializing SwiftNativeFileSystem against URI <br>

<a href="swift://data.os/blob1">swift://data.os/blob1</a> and working dir <a href="swift://data.os/user/ubuntu">

swift://data.os/user/ubuntu</a><br>

2014-02-25 22:10:38,593 DEBUG org.apache.hadoop.fs.swift.http.RestClientBindings: Filesystem

<a href="swift://data.os/blob1">swift://data.os/blob1</a> is using configurati<br>

on keys fs.swift.service.os<br>

2014-02-25 22:10:38,594 DEBUG org.apache.hadoop.fs.swift.http.SwiftRestClient: Service={os} container={data} uri={<a href="http://KEYSTONE:5">http://KEYSTONE:5</a><br>

000/v2.0/tokens} tenant={test} user={tester} region={(none)} publicURL={true} location aware={false} partition size={4718592 KB}, buff<br>

er size={64 KB} block size={32768 KB} connect timeout={15000}, retry count={3} socket timeout={60000} throttle delay={0}<br>

2014-02-25 22:10:38,594 DEBUG org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem: SwiftFileSystem initialized<br>

2014-02-25 22:10:38,599 DEBUG org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore: Reading

<a href="swift://data.os/blob1">swift://data.os/blob1</a> from proxy n<br>

ode<br>

2014-02-25 22:10:38,607 INFO org.apache.hadoop.tools.DistCp: FAIL blob1 : org.apache.hadoop.fs.swift.exceptions.SwiftInternalStateExcep<br>

tion: Null Endpoint -client is not authenticated<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.fs.swift.http.SwiftRestClient.checkNotNull(SwiftRestClient.java:1800)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.fs.swift.http.SwiftRestClient.pathToURI(SwiftRestClient.java:1629)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.fs.swift.http.SwiftRestClient.pathToURI(SwiftRestClient.java:1669)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.fs.swift.http.SwiftRestClient.getData(SwiftRestClient.java:711)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystemStore.getObject(SwiftNativeFileSystemStore.java:294)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.fs.swift.snative.SwiftNativeInputStream.<init>(SwiftNativeInputStream.java:103)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.open(SwiftNativeFileSystem.java:555)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.fs.swift.snative.SwiftNativeFileSystem.open(SwiftNativeFileSystem.java:536)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:436)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.tools.DistCp$CopyFilesMapper.copy(DistCp.java:419)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:547)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.tools.DistCp$CopyFilesMapper.map(DistCp.java:314)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.mapred.Child$4.run(Child.java:255)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at java.security.AccessController.doPrivileged(Native Method)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at javax.security.auth.Subject.doAs(Subject.java:415)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.mapred.Child.main(Child.java:249)<br>

<br>

2014-02-25 22:10:47,700 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceR<br>

etainSize=-1<br>

2014-02-25 22:10:47,723 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:tester cause:java.io.IOExc<br>

eption: Copied: 0 Skipped: 0 Failed: 1<br>

2014-02-25 22:10:47,725 WARN org.apache.hadoop.mapred.Child: Error running child<br>

java.io.IOException: Copied: 0 Skipped: 0 Failed: 1<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.tools.DistCp$CopyFilesMapper.close(DistCp.java:582)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.mapred.Child$4.run(Child.java:255)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at java.security.AccessController.doPrivileged(Native Method)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at javax.security.auth.Subject.doAs(Subject.java:415)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)<br>

<span class="Apple-tab-span" style="white-space:pre"></span>at org.apache.hadoop.mapred.Child.main(Child.java:249)<br>

2014-02-25 22:10:47,739 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task<br>

ubuntu@slave1:/var/log/hadoop/ubuntu/userlogs/job_201402252129_0006/attempt_201402252129_0006_m_000000_3$ <br>

<br>

<div>The hadoop distcp command entered was:</div>

<div><br>

</div>

<div>master$ hadoop distcp -D fs.swift.service.os.public=true -D fs.swift.service.os.tenant=test -D fs.swift.service.os.username=tester -D fs.swift.service.os.password=supersecret

<a href="swift://data.os/blob1">swift://data.os/blob1</a> hdfs://<hdfs ip>:9000/data/blob3</div>

<div><br>

</div>

<div>Am I missing anything obvious?</div>

<div><br>

<div>Thanks and regards,</div>

<div>Ross</div>

<div><br>

</div>

<br>

<br>

<div apple-content-edited="true">--<br>

Ross Lillie<br>

Distinguished Member of Technical Staff<br>

Motorola Solutions, Inc.<br>

<br>

<a href="http://motorolasolutions.com">motorolasolutions.com</a><br>

O: +1.847.576.0012<br>

M: +1.847.980.2241<br>

E: ross.lillie@motorolasolutions.com<br>

<br>

<br>

<span><img height="33" width="277" apple-inline="yes" id="2D4A8761-3848-40D2-89A6-768D3B141F2C" apple-width="yes" apple-height="yes" src="cid:75FF7557-F6FD-4406-A0BB-5CBD9BE3780A@comm.mot.com"></span></div>

<br>

</div>

</body>

</html>