Open Stack

Hi guys!

Yesterday we discussed the problem about additional resources. Our decision
was to use the following structure of JobOrigin:
{
"name":"j_o",
"url":"savanna://3bfae02a-87fb-4f5e-bee5-baa135ff4be0"
"libs":[swift://example.savanna/lib1.jar, hdfs://user/hadoop/lib2.jar]
}

But today I realized that we still need to determine credentials in this
case. So I decided to refuse from different prefixes. After that it looks
like:

{
"name":"j_o",
"url":"swift://example.savanna/job.jar"
"libs":[swift://example.savanna/lib1.jar, swift://example.savanna/lib2.jar]
credentials:{
"user":"admin",
"password":"abacaba"
}
}

Ok. But after a lunch I understood that we need the names to store files to
cluster-hdfs:  in pig scripts user may use REGISTER blablabla.jar.
If we use internal storage we can fetch the name from JobBinary. But what
if it's a swift? My thirst thought was to cut off the name from url. But it
looks like a hack.
I considered the idea of generating senseless names by Savanna. But it is a
strange way to use random string without .jar or .pig extension.

So… What do you think about an idea to create "DataSource" for job
binaries?  We may use even the same model object, only need to rename it.
Result:
for job.jar:
{
    "name": "job.jar",
    "description": "jab file",
    "type": "swift",
  "url": "swift://integration.savanna/input",
    "credentials": {
        "user": "admin",
        "password": "abacaba"
    }
}

for lib1.jar:
{
    "name": "lib.jar",
    "description": "lib which contains …..",
    "type": "internal",
  "url": "savanna-db://3bfae02a-87fb-4f5e-bee5-baa135ff4be0"
}

for lib2.jar:
{
    "name": "lib2.jar",
    "description": "lib which contains …..",
    "type": "hdfs",
  "url": "hdfs://user/hadoop/lib2.jar",
    "credentials": {
        "user": "admin",
        "password": "abacaba"
    }
}

JobOrigin and "JobSource" have many=many relation because we may allow user
to use a lib for different jobs.
I need your comments!

Thanks,
Nadya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20130904/b63ac789/attachment.html>