Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oozie workflow file not found spark job error finding file

Oozie workflow file not found spark job error finding file

New Contributor

I am getting a really a file not found error in my oozie workflow but it is telling me that this file does not exist

insert_hive_all_table1.hql

I have checked the file exists and I have attached the link to the file on hdfs with the file tag <file> ... </file>.

Here is my attempt at the spark job in the oozie workflow.

Any help is appreciated

<start to="spark2-submit"/>
 
  <action name="spark2-submit">
    <shell xmlns="uri:oozie:shell-action:0.3">
      <exec>spark-command.sh</exec>
      <argument>${keyTabLocalPath}</argument>
      <argument>${keyTabUsername}</argument>
      <argument>${kerberos_realm}</argument>
      <argument>spark2-submit --master yarn --deploy-mode cluster --num-executors 6 --executor-memory 8G --driver-memory 8G --driver-cores 6 --conf spark.yarn.queue=${queueName} --conf spark.dynamicAllocation.enabled=false --principal ${edgeUser}@${kerberos_realm} --keytab ${keyTabName} --jars nbd-ingestion-framework.jar --files /etc/hive/conf/hive-site.xml,propagation-conf.xml,loadUsrEnv.sh,insert_hive_all_table1.hql --class GenerateCalFilesChris CalFileGeneration-1.0-SNAPSHOT-jar-with-dependencies.jar
      ${execUserArg} ${execUserKeyTabPathArg} ${configFileArg} insert_hive_all_table1.hql   


	</argument>
      <env-var>HADOOP_CONF_DIR=/etc/hadoop/conf</env-var>
     <file>${projectPath}/Util/spark-command.sh#spark-command.sh</file>
      <file>${keyTabLocation}/${keyTabName}</file>
      <file> /Stuff/Code/DISTCP_CAL/Util/insert_hive_all_table.hql#insert_hive_all_table1.hql</file>
      <file>${nameNode}/${commonPath}/loadUsrEnv.sh#loadUsrEnv.sh</file>
    </shell>
    <ok to="end"/>
    <error to="errorHandler"/>
  </action>


1 REPLY 1

Re: Oozie workflow file not found spark job error finding file

You are changing the file name from insert_hive_all_table.hql adding an extra 1 to the file name insert_hive_all_table1.hql - Is this intended? Also there is an extra space at the beginning of the file element I suggest you remove this as well and test.