0   I am trying run a coordinator job in oozie through ambari gui.. but am getting below exception when i create a dataset input event... ACTION[-] Exception occurred:Unformatted function :2019-11-02T18:10Z Making the job failed I am giving start and end time in gui... <?xml version="1.0" encoding="UTF-8" standalone="no"?><coordinator-app xmlns="uri:oozie:coordinator:0.5" end="2019-11-01T18:20Z" frequency="${coord:days(1)}" name="ETLTest2" start="2019-11-01T18:15Z" timezone="GMT-05:00"> <datasets> <dataset frequency="${coord:days(1)}" initial-instance="2019-11-01T18:15Z" name="ETL_MDM" timezone="GMT-06:00"> <uri-template>hdfs://devnamesvccluster/user/sparkcronsvc/edm_ucsys/ingestion/</uri-template> </dataset> </datasets> <input-events> <data-in dataset="ETL_MDM" name="ETL_Check_Sqoop"> <start-instance>2019-11-02T18:10Z</start-instance> <end-instance>2019-11-01T18:30Z</end-instance> </data-in> </input-events> <action> <workflow> <apppath>/user/spid/edm_tab/full/workflow_etl2.xml</app-path> </workflow> </action> </coordinator-app> Let me know if i missing something... ... View more

I was also try to do [something similar to the action described in this thread]… to use it use in shell action but am getting error sqoop command not found when running as shell action. But strange sqoop action with args work…but don't know how i can store the value of sqoop eval in sqoop action. ... View more

Actually I have to start a sqoop action in oozie  based on the condition that ETL job is completed. ETL job gets completed when one table has last row with success in one column. So when we sees that value in Sql server database we can start the sqoop action job. Ow Sqoop has to wait and look for that thing again and once it sees the value .. it starts the process.   What is the best strategy to achieve this? ... View more

I am running a shell script of sqoop through oozie which takes a list of tables/schema from list file in the script and load the data in data lake in kerberos cluster. For running script we pass servername databasename as a argument. It runs well in edge node but failing in oozie workflow with yarn logs as sqoop command not found. Script sqoop_full.sh is given below: #!/usr/bin/bash dmserver=$1 hdp_dir=$2 currDir=`realpath ${0} | rev | cut -d"/" -f2- | rev` while read -r line do tbl=`echo $line | cut -d "|" -f1` dmdb=`echo $line | cut -d "|" -f2` sqoop import -Dmapreduce.job.user.classpath.first=true -Dhadoop.security.credential.provider.path=jceks://hdfs/user/root/mssql.datamarthdpservice.pass.jceks \ --connect 'jdbc:jtds:sqlserver://'${dmserver}'.use.ucdp.net/'${dmdb}';integratedSecurity=true;domain=use.ucdp.net;database='${dmdb}';' \ --username datamhdpser \ --password-alias mssql.datamarthdpservice.pass \ --query "SELECT * from ${dmdb}.dbo.${tbl} WITH(NOLOCK) WHERE \$CONDITIONS" -m 1 \ --as-avrodatafile \ --compression-codec snappy \ --delete-target-dir \ --target-dir="hdfs:///data/collection/${hdp_dir}/${tbl}" 2>&1| tee -a ${currDir}/logs/output.log done < <(tail -n +2 fulllist.txt | grep -v "#")   Workflow.xml is : <?xml version="1.0" encoding="UTF-8" standalone="no"?> <workflow-app xmlns="uri:oozie:workflow:0.5" name="Edm_Sqoop_full"> <start to="shell_1"/> <action name="shell_1"> <shell xmlns="uri:oozie:shell-action:0.3"> <job-tracker>${resourceManager}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>oozie.use.system.libpath</name> <value>true</value> </property> <property> <name>oozie.libpath</name> <value>//user/oozie/share/lib/lib_20190213172012/sqoop</value> </property> <property> <name>hadoop.security.credential.provider.path</name> <value>jceks://hdfs/user/root/mssql.datamarthdpservice.pass.jceks</value> </property> <property> <name>mapreduce.user.classpath.first</name> <value>true</value> </property> <property> <name>mapreduce.task.classpath.user.precedence</name> <value>true</value> </property> </configuration> <exec>/user/sparkcronsvc/edm/sqoop_full.sh</exec> <argument>uso12smktdb01</argument> <argument>unica</argument> <file>/user/sparkcronsvc/edm/sqoop_full.sh</file> <file>/user/sparkcronsvc/edm/fulllist.txt</file> </shell> <ok to="end"/> <error to="kill"/> </action> <kill name="kill"> <message>${wf:errorMessage(wf:lastErrorNode())}</message> </kill> <end name="end"/> </workflow-app>   Yarn logs: >>> Invoking Main class now >>> Setting [tez.application.tags] tag: oozie-452d4388e731e788275adcc4b7f33c1d Current working dir /data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002 Full Command .. ------------------------- 0:sqoop_full.sh: 1:uso12smktdb01: 2:unica: List of passing environment ------------------------- PATH=.:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent: HADOOP_CONF_DIR=/usr/hdp/2.6.4.125-1/hadoop/conf: STDOUT_LOGFILE_ENV=/data/disk15/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/stdout: HADOOP_ROOT_LOGGER=INFO,console: JAVA_HOME=/usr/java/latest: LANG=en_US.UTF-8: NM_HOST=usor7dhdpw01.use.ucdp.net: HADOOP_CLASSPATH=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/*:/data/disk8/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000001:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data/disk8/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000001/*:null: LD_LIBRARY_PATH=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:/usr/hdp/2.6.4.125-1/hadoop/lib/native:/usr/hdp/2.6.4.125-1/hadoop/lib/native/Linux-amd64-64:./mr-framework/hadoop/lib/native:./mr-framework/hadoop/lib/native/Linux-amd64-64: STDERR_LOGFILE_ENV=/data/disk15/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/stderr: LOGNAME=sparkcronsvc: JVM_PID=23287: PWD=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002: HADOOP_CLIENT_OPTS=: _=/usr/java/latest/bin/java: LOCAL_DIRS=/data/disk1/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk2/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk3/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk4/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk5/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk6/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk7/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk8/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk9/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk10/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk11/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk12/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk13/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk14/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk15/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk16/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk18/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk19/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk20/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk21/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk22/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk23/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk24/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477: SHELL=/bin/bash: NM_HTTP_PORT=8042: LOG_DIRS=/data/disk1/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk2/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk3/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk4/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk5/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk6/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk7/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk8/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk9/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk10/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk11/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk12/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk13/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk14/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk15/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk16/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk17/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk18/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk19/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk20/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk21/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk22/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk23/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk24/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002: PRELAUNCH_OUT=/data/disk1/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/prelaunch.out: NM_AUX_SERVICE_mapreduce_shuffle=AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA= : NM_PORT=45454: HADOOP_YARN_HOME=/usr/hdp/2.6.4.125-1/hadoop-yarn: USER=sparkcronsvc: CLASSPATH=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/mapreduce/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/common/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/common/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/yarn/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/yarn/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/hdfs/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.4.125-1/hadoop/lib/hadoop-lzo-0.6.0.2.6.4.125-1.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/*:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/*: PRELAUNCH_ERR=/data/disk1/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/prelaunch.err: HADOOP_TOKEN_FILE_LOCATION=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/container_tokens: NM_AUX_SERVICE_spark_shuffle=: LOCAL_USER_DIRS=/data/disk1/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk2/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk3/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk4/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk5/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk6/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk7/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk8/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk9/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk10/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk11/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk12/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk13/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk14/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk15/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk16/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk18/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk19/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk20/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk21/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk22/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk23/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk24/hadoop/yarn/local/usercache/sparkcronsvc/: HADOOP_HOME=/usr/hdp/2.6.4.125-1/hadoop: OOZIE_ACTION_CONF_XML=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/action.xml: SHLVL=1: HOME=/home/: NM_AUX_SERVICE_spark2_shuffle=: CONTAINER_ID=container_e45_1559931840942_7477_01_000002: MALLOC_ARENA_MAX=4: ================================================================= >>> Invoking Shell command line now >> Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found ... View more