Open Stack

Answers to other questions:

2) (first part) Yes, I think Oozie shell actions are a great idea. I can
help work on a spec for this.

In general, Sahara should be able to support any kind of Oozie action.
Each will require a new job type, changes to the Oozie engine, and a UI
form to handle submission. We talked about shell actions once upon a
time. I don't think a spec for that will be too difficult.

Typically when adding new Oozie actions, I start by running things with
the Oozie command line to figure out what's possible and what the
workflow.xml looks like in general.


We also talked about allowing a user to upload raw workflows -- the
difficulty there is figuring out what Sahara generates vs what the user
writes, so this may be a more complicated topic. I think it will have to
wait for another cycle.

2) (error information)

Yes, the lack of good error information is a big problem in my opinion,
but we have no plan for it at this time.

The OpenStack approach seems to be to look through lots of log files to
identify errors.  For EDP, we may need to support a similar approach by
allowing job logs to be easily retrieved from clusters and written
somewhere a user can parse through them for error information.  Any
ideas on how to do this are welcome.

Trevor

-- 

(2)     Sahara oozie plan

So when I search the solution for HBase test case, I found
http://archive.cloudera.com/cdh5/cdh/5/oozie/DG_ShellActionExtension.html , it talks about oozie shell action job type, I believe my first issue in EDP job in java action can be solved by shell action, because I can set <exec>java</exec> <argument>`hbase classpath`</argument> in workflow.xml, just like the way I run this jar in the vm console by command. So I raise a bp for adding oozie shell action https://blueprints.launchpad.net/sahara/+spec/add-edp-shell-action  I will make further research on the bp/specs and update the spec. In today’s meeting , you mentioned about allow user to upload his own workflow.xml, I am interesting about this , we can provide our support to this part, so can you provide some bp/specs or other docs for me? So we can discuss for more.

For more, is there any plan to provide edp job error info to the user? I
think this is also important, currently we just have "killed" label, no
more information.