Open Stack

Hi,

  Thanks for your patience.  I have been consumed with spark-swift, but
I can start to address these questions now :)

On (1) (a) below, I will try to reproduce and look at how we can better
support classpath in EDP. I'll let you know what I find.
We may need to add some configuration options for EDP or change how it
works.

On (1) (b) below, in the edp-move-examples.rst spec for Juno we
described a directory structure that could be used
for separating hadoop1 vs hadoop2 specific directories.  Maybe we can do
something similar based on plugins

For instance, if we have some hbase examples, we can make subdirectories
for each plugin.  Common parts can be
shared, plugin-specific files can be stored in the subdirectories.

(and perhaps the "hadoop2" example already there should just be a
subdirectory under "edp-java")

Best,

Trevor

--

Hi McKay
Thx for your support
I will talk details of these items as below:

(1)     EDP job in Java action

   The background is that we want write integration test case for newly
added services like HBase, zookeeper just like the way the edp-examples
does( sample code under sahara/etc/edp-examples/). So I thought I can
wrote an example via edp job by Java action to test HBase Service, then
I wrote the HBaseTest.java and packaged as a jar file, and run this jar
manually with the command "java -cp `hbase classpath` HBaseTest.jar
HBaseTest", it works well in the vm(provisioned by sahara with cdh
plugin).
“/usr/lib/jvm/java-7-oracle-cloudera/bin/java -cp "HBaseTest.jar:`hbase
classpath`" HBaseTest”
So I want run this job via horizon in sahara job execution page, but
found no place to pass the `hbase classpath` parameter.(I have tried
java_opt and configuration and args, all failed). When I pass the “-cp
`hbase classpath`” to java_opts in horizon job execution page. Oozie
raise this error as below

“2015-01-15 16:43:26,074 WARN
org.apache.oozie.action.hadoop.JavaActionExecutor:
SERVER[hbase-master-copy-copy-001.novalocal] USER[hdfs] GROUP[-] TOKEN[]
APP[job-wf] JOB[0000045-150105050354389-oozie-oozi-W]
ACTION[0000045-150105050354389-oozie-oozi-W at job-node] LauncherMapper
died, check Hadoop LOG for job
[hbase-master-copy-copy-001.novalocal:8032:job_1420434100219_0054]
2015-01-15 16:43:26,172 INFO
org.apache.oozie.command.wf.ActionEndXCommand:
SERVER[hbase-master-copy-copy-001.novalocal] USER[hdfs] GROUP[-] TOKEN[]
APP[job-wf] JOB[0000045-150105050354389-oozie-oozi-W]
ACTION[0000045-150105050354389-oozie-oozi-W at job-node] ERROR is
considered as FAILED for SLA”

So I stuck with this issue, I can’t write the integration test in sahara
( could not pass the classpath parameter), 
I have check oozie official site .
https://cwiki.apache.org/confluence/display/OOZIE/Java+Cookbook found no
help info.
 
   So about the EDP job in java, I have two problems right now:
a)      How to pass classpath to java action as I mention before. So
this also reminds me that we can allow user to modify or upload this own
workflow.xml, then we can provide more options for user.
b)      I concern that it’s hard to have a common edp-example for HBase
for all plugin(cdh dhp), because the example code depends on third party
jars(for example hbase-client.jar…) and different platform(CDH HDP) they
may have different version hbase-client.jar, for example, cdh use
hbase-client-0.98.6-cdh5.2.1.jar. 

attached is a zip file which contains HBaseTest.jar and the source code.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150202/9768d0f0/attachment.html>