Member since
10-17-2016
11
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
813 | 02-23-2017 09:56 AM |
06-28-2017
01:05 PM
Hi All,
I am trying to run a spark action in oozie and getting Null Pointer exception. Below is my workflow.xml and exception details...
workflow.xml
<workflow-app xmlns="uri:oozie:workflow:0.5" name="Spark_Test">
<global>
<job-tracker>${job_tracker}</job-tracker>
<name-node>${name_node}</name-node>
</global>
<credentials>
<credential name="hiveCredentials" type="hive2">
<property>
<name>hive2.jdbc.url</name>
<value>${hive_beeline_server}</value>
</property>
<property>
<name>hive2.server.principal</name>
<value>${hive_kerberos_principal}</value>
</property>
</credential>
</credentials>
<start to="SparkTest" />
<action name="SparkTest" cred="hiveCredentials">
<spark xmlns="uri:oozie:spark-action:0.1">
<job-tracker>${job_tracker}</job-tracker>
<name-node>${name_node}</name-node>
<master>yarn-client</master>
<name>Spark Hive Example</name>
<class>com.fbr.genjson.exec.GenExecJson</class>
<jar>${jarPath}/fedebomrpt_genjson.jar</jar>
<spark-opts>--jars /usr/hdp/current/spark-client/lib/datanucleus-api-jdo-3.2.6.jar,/usr/hdp/current/spark-client/lib/datanucleus-rdbms-3.2.9.jar,/usr/hdp/current/spark-client/lib/datanucleus-core-3.2.10.jar --files /etc/hive/conf/hive-site.xml --conf spark.sql.hive.convertMetastoreOrc=false --driver-memory 2g --executor-memory 16g --executor-cores 4 --conf spark.ui.port=5051 --queue fbr</spark-opts>
<arg>${arg1}</arg>
<arg>${arg2}</arg>
</spark>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>Spark Java PatentCitation failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
Exception:
SERVER[xxx.hpc.xx.com] USER[prxtcbrd] GROUP[-] TOKEN[] APP[Spark_Test] JOB[0004629-170625082345353-oozie-oozi-W] ACTION[0004629-170625082345353-oozie-oozi-W@SparkTest] Error starting action [SparkTest]. ErrorType [ERROR], ErrorCode [NullPointerException], Message [NullPointerException: null]
org.apache.oozie.action.ActionExecutorException: NullPointerException: null
at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:446)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1202)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1373)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
at org.apache.oozie.action.hadoop.SparkActionExecutor.setupActionConf(SparkActionExecutor.java:85)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1091)
... 11 more
... View more
03-30-2017
11:15 AM
Hi All,We are using HDP hdp/2.4.2.0-258 version and sqoop version: sqoop-1.4.6.2.4.2.0-258 I am trying for sqoop incremental import, for that I need to connect to metastore to create sqoop jobs. But I couldnt find the sqoop-site.xml in /usr/hdp/current/sqoop-client/conf .. Can anyone point out the exact path.
... View more
Labels:
03-30-2017
09:15 AM
Can we use variables inside sqoop commands while creating sqoop jobs and passing the values as parameters to the job.
For example something like below
sqoop --create myjob -- import --connect jdbc:xxx.. --username $username --password $Password ..
And pass the arguments while executing the job. sqoop job -exec myjob username password
... View more
Labels:
03-10-2017
10:37 AM
It worked.. Thanks..
... View more
03-09-2017
06:48 AM
Hi All, I am trying to split a string with semicolon using the below query. In Beeline, Its thrwing me Parse execption error. But in Hive its working fine. I am properly escaping the semicolan in my query. Below is my query. select split(fieldname,'\;') from table name I also tried select split(fieldname,'\\;') from table name I also tried with double quotes. But Nothing is working in Beeline shell. But in Hive shell, no issues. Query is running fine. Beeline is throwing below error Error: Error while compiling statement: FAILED: ParseException line 1:29 cannot recognize input near '<EOF>' '<EOF>' '<EOF>' in select expression (state=42000,code=40000) Kindly let me know the proper way to escape the semi colans.
... View more
Labels:
02-23-2017
09:56 AM
Adding the below Credential properties fixed my issue. <credentials>
<credential name="hbase-cred" type="hbase">
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hbase.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hbase.master.kerberos.principal</name>
<value>hbase/_HOST@XX.XX.COM</value>
</property>
<property>
<name>hbase.regionserver.kerberos.principal</name>
<value>hbase/_HOST@XX.XX.COM</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>${zookeeper_quorum}</value>
</property>
<property>
<name>hadoop.rpc.protection</name>
<value>authentication</value>
</property>
<property>
<name>hbase.rpc.protection</name>
<value>authentication</value>
</property>
<property>
<name>hbase.master.keytab.file</name>
<value>/etc/hadoop/hbase.keytab</value>
</property>
<property>
<name>hbase.regionserver.keytab.file</name>
<value>/etc/hadoop/hbase.keytab</value>
</property>
</credential>
</credentials>
... View more
02-22-2017
08:04 AM
@Ajay I am not an admin to do this. I believe sharelib is properly set. Hive Actions are running fine.
... View more
02-21-2017
05:49 PM
I'm running a test hbase java program via oozie java action. The following error is encountered : Error,
java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
at org.apache.oozie.action.hadoop.HbaseCredentials.copyHbaseConfToJobConf(HbaseCredentials.java:67)
at org.apache.oozie.action.hadoop.HbaseCredentials.addtoJobConf(HbaseCredentials.java:55)
at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1287)
at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1124)
at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1373)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232)
at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
at org.apache.oozie.command.XCommand.call(XCommand.java:287)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331)
at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745) The program runs correctly from command line : java -cp `hbase classpath` personal_project.HBaseConnector Is there a way I can pass output of 'hbase classpath' to the oozie java action. I have copied below hbase jars to workflow's lib directory as well.
hbase-common-$HBASE_VERSION.jar hbase-client-$HBASE_VERSION.jar hbase-server-$HBASE_VERSION.jar hbase-protocol-$HBASE_VERSION.jar hbase-hadoop2-compat-$HBASE_VERSION.jar htrace-core-*.jar hive-hbase-handler-$HBASE_VERSION.jar Following is the java action from workflow.xml : <workflow-app xmlns="uri:oozie:workflow:0.5" name="hbase_connect">
<global>
<job-tracker>${job_tracker}</job-tracker>
<name-node>${name_node}</name-node>
</global>
<credentials>
<credential name="hbase-cred" type="hbase">
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hbase.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hbase.master.kerberos.principal</name>
<value>hbase/_HOST@XX.XX.COM</value>
</property>
<property>
<name>hbase.regionserver.kerberos.principal</name>
<value>hbase/_HOST@XX.XX.COM</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<value>${zookeeper_quorum}</value>
</property>
<property>
<name>hadoop.rpc.protection</name>
<value>authentication</value>
</property>
<property>
<name>hbase.rpc.protection</name>
<value>authentication</value>
</property>
</credential>
</credentials>
<start to="hbase_connect_java" />
<action name="hbase_connect_java" cred="hbase-cred">
<java>
<job-xml>${runconfig_dir}/hbase-site.xml</job-xml>
<main-class>personal_project.HBaseConnector</main-class>
<java-opts>-Dsun.security.krb5.debug=true</java-opts>
<arg>${name_node}</arg>
<arg>${kerberos_principal}</arg>
<arg>${kerberos_keytab_file}</arg>
<arg>a:xxx,b:yyy,c:zzz,d:yyy</arg>
<arg>2016-02-22-123456</arg>
<file>${path_hdfs_common}/lib/hbase_connect.jar#hbase_connect.jar</file>
<file>${runconfig_dir}/jaas.conf#jaas.conf</file>
<file>${runconfig_dir}/krb5.conf#krb5.conf</file>
<file>${runconfig_dir}/hbase-site.xml#hbase-site.xml</file>
</java>
<ok to="end"/>
<error to="fail"/>
</action>
<kill name="fail">
<message>${wf:name()} workflow failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<end name="end"/>
</workflow-app>
... View more
Labels:
12-01-2016
06:31 AM
@Pavel Benes Did you fixed the hive issues in Java Action from oozie ?
... View more
10-17-2016
07:48 AM
1 Kudo
Hi All, I have developed Java application to connect to Ldap server and get the
details in csv format based on user arguments. The result csv file will be saved
in HDFS (Hadoop File system). In order to write to HDFS, I have imported
org.apache.hadoop.security.UserGroupInformation and set the
Kerberos configuration properly. Below are the code
snippet. config.set("hadoop.security.authentication",
"Kerberos");UserGroupInformation.setConfiguration(config);
UserGroupInformation.loginUserFromKeytab(Principal,KeyTabfile); we kept the keytab file in edge server of the POC environment. When I run
the Java application from edge server, It is reading the keytab file running
fine and writing the results to HDFS. But my issue starts when I tried to shedule this application using
oozie. Oozie will launch java actions in any of the data nodes
in a cluster based on the availability of resources and oozie cant access the
edge server. Because of this, my java action in oozie getting failed with
security exception since it cant read the keytab file in edge server. Kindly suggest me the solution for this issue
... View more
Labels:
10-17-2016
07:27 AM
@Pavel Benes Did you get the solution. I am facing the similar issue. My java application writes the result to HDFS and it needs kerberos authentication. When schedule my application using oozie. I am facing this issue.
... View more