<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 TRANSITIONAL//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; CHARSET=UTF-8">
<META NAME="GENERATOR" CONTENT="GtkHTML/4.8.5">
</HEAD>
<BODY>
Hi folks,<BR>
<BR>
Here is another way to do this. Lu had mentioned Oozie shell actions previously.<BR>
Sahara doesn't support them, but I played with it from the Oozie command line<BR>
to verify that it solves our hbase problem, too.<BR>
<BR>
We can potentially create a blueprint to build a simple Shell action around a<BR>
user-supplied script and supporting files. The script and files would all be stored<BR>
in Sahara as job binaries (Swift or internal db) and referenced the same way. The exec<BR>
target must be on the path at runtime, or included in the working dir.<BR>
<BR>
To do this, I simply put workflow.xml, doit.sh, and the test jar into<BR>
a directory in hdfs. Then I ran it with the Oozie cli using the job.xml config file<BR>
configured to point at the hdfs dir. Nothing special here, just standard Oozie<BR>
job execution.<BR>
<BR>
I've attached everything here but the test jar.<BR>
<BR>
$ oozie job -oozie http://localhost:11000/oozie -config job.xml -run<BR>
<BR>
Best,<BR>
<BR>
Trev<BR>
<BR>
On Thu, 2015-02-12 at 08:39 -0500, Trevor McKay wrote:<BR>
<BLOCKQUOTE TYPE=CITE>
Hi Lu, folks,<BR>
<BR>
I've been investigating how to run Java actions in Sahara EDP that depend on <BR>
HBase libraries (see snippet from the original question by Lu below).<BR>
<BR>
In a nutshell, we need to use Oozie sharelibs for this. I am working on a spec now, thinking <BR>
about the best way to support this in Sahara, but here is a semi-manual intermediate solution<BR>
that will work if you would like to run such a job from Sahara.<BR>
<BR>
1) Create your own Oozie sharelib that contains the HBase jars.<BR>
<BR>
This ultimately is just an HDFS dir holding the jars. On any node in your cluster with <BR>
HBase installed, run the attached script or something like it (I like Python better than bash :) )<BR>
It simply separates the classpath and uploads all the jars to the specified HDFS dir.<BR>
<BR>
$ parsePath.py /user/myhbaselib<BR>
<BR>
2) Run your Java action from EDP, but use the oozie.libpath configuration value when you<BR>
launch the job. For example, on the job configure tab set oozie.libpath like this:<BR>
<BR>
Name Value<BR>
<BR>
oozie.libpath hdfs://namenode:8020/user/myhbaselib<BR>
<BR>
(note, support for this was added in <A HREF="https://review.openstack.org/#/c/154214/">https://review.openstack.org/#/c/154214/</A>)<BR>
<BR>
That's it! In general, you can add any jars that you want to a sharelib and then set the<BR>
oozie.libpath for the job to access them.<BR>
<BR>
Here is a good blog entry about sharelibs and extra jars in Oozie jobs:<BR>
<BR>
<A HREF="http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/">http://blog.cloudera.com/blog/2014/05/how-to-use-the-sharelib-in-apache-oozie-cdh-5/</A><BR>
<BR>
Best,<BR>
<BR>
Trevor<BR>
<BR>
--- original question<BR>
(1) EDP job in Java action<BR>
<BR>
The background is that we want write integration test case for newly added services like HBase, zookeeper just like the way the edp-examples does( sample code under sahara/etc/edp-examples/). So I thought I can wrote an example via edp job by Java action to test HBase Service, then I wrote the HBaseTest.java and packaged as a jar file, and run this jar manually with the command "java -cp `hbase classpath` HBaseTest.jar HBaseTest", it works well in the vm(provisioned by sahara with cdh plugin). <BR>
“/usr/lib/jvm/java-7-oracle-cloudera/bin/java -cp "HBaseTest.jar:`hbase classpath`" HBaseTest”<BR>
So I want run this job via horizon in sahara job execution page, but found no place to pass the `hbase classpath` parameter.(I have tried java_opt and configuration and args, all failed). When I pass the “-cp `hbase classpath`” to java_opts in horizon job execution page. Oozie raise this error as below<BR>
<BR>
<BR>
<PRE>
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: <A HREF="mailto:OpenStack-dev-request@lists.openstack.org">OpenStack-dev-request@lists.openstack.org</A>?subject:unsubscribe
<A HREF="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</A>
</PRE>
</BLOCKQUOTE>
<BR>
</BODY>
</HTML>