<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>Clay-</div>
<div><br>
</div>
<div>You are correct – the hadoop-swift filesystem is just an implement of the Hadoop FileSystem class that uses swift for storage. As such, it's just translating filesystem calls into HTTP requests into swift and is not accessing the filesystem of the object
servers directly. </div>
<div><br>
</div>
<div>Shashank-</div>
<div><br>
</div>
<div>I'm curious about what version of that Hadoop-Swift-integration project you're trying to run. You shouldn't have been able to create containers with it in any of the more recent versions. You might want to try the "converged" branch of <a href="https://github.com/hortonworks/Hadoop-and-Swift-integration">https://github.com/hortonworks/Hadoop-and-Swift-integration</a>.
This is the branch that's getting submitted back to Apache for inclusion in Hadoop.</div>
<div><br>
</div>
<div>Hope that helps,</div>
<div>-David</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Vaidy Gopalakrishnan <<a href="mailto:gvaidy@gmail.com">gvaidy@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Friday, April 26, 2013 4:06 PM<br>
<span style="font-weight:bold">To: </span>Clay Gerrard <<a href="mailto:clay.gerrard@gmail.com">clay.gerrard@gmail.com</a>>, David Dobbins <<a href="mailto:david.dobbins@rackspace.com">david.dobbins@rackspace.com</a>><br>
<span style="font-weight:bold">Cc: </span>Shashank Sahni <<a href="mailto:shredder12@gmail.com">shredder12@gmail.com</a>>, "<a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a>" <<a href="mailto:openstack@lists.launchpad.net">openstack@lists.launchpad.net</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack] Swift container's rwx permissions<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr"><br>
<div style="">[Hi Clay]</div>
<div style=""><br>
</div>
<div style="">Including David who can answer this much better.</div>
<div style=""><br>
</div>
<div style="">Vaidy</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, Apr 26, 2013 at 12:30 PM, Clay Gerrard <span dir="ltr">
<<a href="mailto:clay.gerrard@gmail.com" target="_blank">clay.gerrard@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Wow, so I glanced quickly at the github project to try and get some context, but I think I actually start getting *more* confused when I see "swift" in the same class name as "file-system" ;)
<div><br>
</div>
<div>I'd like you (or maybe vaidy, hi vaidy!) to correct me if I'm wrong, but this hadoop integration will *not* access the filesystem of the object servers directly? Everything will happen on a pool of processing boxes that will talk to swift via HTTP - same
as any other client?</div>
<div><br>
</div>
<div>In that case, the error message is just a leaky abstraction showing through. HDFS probably has permission errors that it tries to helpfully map back to file system constructs which just don't apply when you're trying to "simulate" a file system on object
storage. You'll have to get down the the HTTP to understand what's causing the error. Presumably a 401 from Swift, so access to swift logs would be helpful.</div>
<div><br>
</div>
<div>OTOH, if we're * actually* talking about filesystem permissions; then I'm totally confused. But ACL's definitely won't help. They're just a row sitting in a sqlite database - probably on a totally different server from where the one replica of this object
is sitting on the filesystem. Nothing you can set in the api will change the filesystem permissions of the directory structure or files on the object servers.</div>
<div><br>
</div>
<div>Maybe do you have some more overview info on the general approach? I don't really have any Hadoop experience, so maybe it'd be better if there's a hadoop expert out there that also has some experience with swift and can help get you on the right track...</div>
<div><br>
</div>
<div>-Clay</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>In </div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, Apr 26, 2013 at 1:11 AM, Shashank Sahni <span dir="ltr">
<<a href="mailto:shredder12@gmail.com" target="_blank">shredder12@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>
<div>
<div>
<div>Hi everyone,<br>
<br>
</div>
I've been experimenting with using Swift(1.8) as Hadoop's DFS backend. <br>
<br>
<a href="https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration" target="_blank">https://github.com/DmitryMezhensky/Hadoop-and-Swift-integration</a><br>
<br>
</div>
After a few glitches, I'm able to create/access/delete objects/containers using hadoop's cli fs tool. But whenever I'm trying to run the job it fails with the following error.<br>
<br>
ERROR security.UserGroupInformation: PriviledgedActionException as:dharmesh cause:java.io.IOException: The ownership/permissions on the staging directory swift://hadooptest.rackspace/tmp/app/mapred/staging/dharmesh/.staging is not as expected. It is owned by
and permissions are rwxrwxrwx. The directory must be owned by the submitter dharmesh or by dharmesh and permissions must be rwx------<br>
<br>
</div>
Note that, the local system username is "dharmesh" and the openstack account and associated tenant is "dharmesh" too.<br>
<br>
</div>
I tried setting the permissions by creating "tmp" container using "swift post -r 'dharmesh:dharmesh'", but unfortunately ended up with the same result. Is there an other way to set rwx ACLs in swift?<span class="HOEnZb"><font color="#888888"><br>
<div>
<div><br>
<div>
<div>
<div>
<div>--<br>
Shashank Sahni<br>
</div>
</div>
</div>
</div>
</div>
</div>
</font></span></div>
<span class="HOEnZb"><font color="#888888"><br>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br>
</font></span></blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</span>
</body>
</html>