Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

oozie shell action : sqoop command not found

oozie shell action : sqoop command not found

Explorer

I am running a shell script of sqoop through oozie which takes a list of tables/schema from list file in the script and load the data in data lake in kerberos cluster.

For running script we pass servername databasename as a argument.

It runs well in edge node but failing in oozie workflow with yarn logs as sqoop command not found.

Script sqoop_full.sh is given below:

#!/usr/bin/bash
dmserver=$1
hdp_dir=$2

currDir=`realpath ${0} | rev | cut -d"/" -f2- | rev`

while read -r line
do
tbl=`echo $line | cut -d "|" -f1`
dmdb=`echo $line | cut -d "|" -f2`

sqoop import -Dmapreduce.job.user.classpath.first=true -Dhadoop.security.credential.provider.path=jceks://hdfs/user/root/mssql.datamarthdpservice.pass.jceks \
--connect 'jdbc:jtds:sqlserver://'${dmserver}'.use.ucdp.net/'${dmdb}';integratedSecurity=true;domain=use.ucdp.net;database='${dmdb}';' \
--username datamhdpser \
--password-alias mssql.datamarthdpservice.pass \
--query "SELECT * from ${dmdb}.dbo.${tbl} WITH(NOLOCK) WHERE \$CONDITIONS" -m 1 \
--as-avrodatafile \
--compression-codec snappy \
--delete-target-dir \
--target-dir="hdfs:///data/collection/${hdp_dir}/${tbl}" 2>&1| tee -a ${currDir}/logs/output.log

done < <(tail -n +2 fulllist.txt | grep -v "#")

 

Workflow.xml is :

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<workflow-app xmlns="uri:oozie:workflow:0.5" name="Edm_Sqoop_full">
<start to="shell_1"/>
<action name="shell_1">
<shell xmlns="uri:oozie:shell-action:0.3">
<job-tracker>${resourceManager}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
<property>
<name>oozie.use.system.libpath</name>
<value>true</value>
</property>
<property>
<name>oozie.libpath</name>
<value>//user/oozie/share/lib/lib_20190213172012/sqoop</value>
</property>
<property>
<name>hadoop.security.credential.provider.path</name>
<value>jceks://hdfs/user/root/mssql.datamarthdpservice.pass.jceks</value>
</property>
<property>
<name>mapreduce.user.classpath.first</name>
<value>true</value>
</property>
<property>
<name>mapreduce.task.classpath.user.precedence</name>
<value>true</value>
</property>
</configuration>
<exec>/user/sparkcronsvc/edm/sqoop_full.sh</exec>
<argument>uso12smktdb01</argument>
<argument>unica</argument>
<file>/user/sparkcronsvc/edm/sqoop_full.sh</file>
<file>/user/sparkcronsvc/edm/fulllist.txt</file>
</shell>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>${wf:errorMessage(wf:lastErrorNode())}</message>
</kill>
<end name="end"/>
</workflow-app>

 

Yarn logs:

>>> Invoking Main class now >>>

Setting [tez.application.tags] tag: oozie-452d4388e731e788275adcc4b7f33c1d
Current working dir /data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002
Full Command .. 
-------------------------
0:sqoop_full.sh:
1:uso12smktdb01:
2:unica:
List of passing environment
-------------------------
PATH=.:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:
HADOOP_CONF_DIR=/usr/hdp/2.6.4.125-1/hadoop/conf:
STDOUT_LOGFILE_ENV=/data/disk15/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/stdout:
HADOOP_ROOT_LOGGER=INFO,console:
JAVA_HOME=/usr/java/latest:
LANG=en_US.UTF-8:
NM_HOST=usor7dhdpw01.use.ucdp.net:
HADOOP_CLASSPATH=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/*:/data/disk8/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000001:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data/disk8/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000001/*:null:
LD_LIBRARY_PATH=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:/usr/hdp/2.6.4.125-1/hadoop/lib/native:/usr/hdp/2.6.4.125-1/hadoop/lib/native/Linux-amd64-64:./mr-framework/hadoop/lib/native:./mr-framework/hadoop/lib/native/Linux-amd64-64:
STDERR_LOGFILE_ENV=/data/disk15/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/stderr:
LOGNAME=sparkcronsvc:
JVM_PID=23287:
PWD=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:
HADOOP_CLIENT_OPTS=:
_=/usr/java/latest/bin/java:
LOCAL_DIRS=/data/disk1/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk2/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk3/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk4/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk5/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk6/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk7/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk8/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk9/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk10/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk11/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk12/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk13/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk14/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk15/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk16/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk18/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk19/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk20/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk21/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk22/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk23/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk24/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477:
SHELL=/bin/bash:
NM_HTTP_PORT=8042:
LOG_DIRS=/data/disk1/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk2/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk3/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk4/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk5/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk6/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk7/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk8/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk9/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk10/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk11/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk12/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk13/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk14/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk15/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk16/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk17/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk18/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk19/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk20/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk21/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk22/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk23/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk24/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:
PRELAUNCH_OUT=/data/disk1/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/prelaunch.out:
NM_AUX_SERVICE_mapreduce_shuffle=AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
:
NM_PORT=45454:
HADOOP_YARN_HOME=/usr/hdp/2.6.4.125-1/hadoop-yarn:
USER=sparkcronsvc:
CLASSPATH=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/mapreduce/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/common/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/common/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/yarn/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/yarn/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/hdfs/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.4.125-1/hadoop/lib/hadoop-lzo-0.6.0.2.6.4.125-1.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/*:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/*:
PRELAUNCH_ERR=/data/disk1/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/prelaunch.err:
HADOOP_TOKEN_FILE_LOCATION=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/container_tokens:
NM_AUX_SERVICE_spark_shuffle=:
LOCAL_USER_DIRS=/data/disk1/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk2/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk3/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk4/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk5/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk6/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk7/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk8/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk9/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk10/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk11/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk12/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk13/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk14/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk15/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk16/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk18/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk19/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk20/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk21/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk22/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk23/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk24/hadoop/yarn/local/usercache/sparkcronsvc/:
HADOOP_HOME=/usr/hdp/2.6.4.125-1/hadoop:
OOZIE_ACTION_CONF_XML=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/action.xml:
SHLVL=1:
HOME=/home/:
NM_AUX_SERVICE_spark2_shuffle=:
CONTAINER_ID=container_e45_1559931840942_7477_01_000002:
MALLOC_ARENA_MAX=4:
=================================================================

>>> Invoking Shell command line now >>

Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
2 REPLIES 2

Re: oozie shell action : sqoop command not found

Community Manager

@arunskumar You probably have multiple things going wrong here, but as a first step it won't hurt to modify your bash shell script so that on line 12:

 


sqoop import -Dmapreduce.job.user.classpath.first=true 
…more command line flags, etc.…
--target-dir="hdfs:///data/collection/${hdp_dir}/${tbl}" 2>&1| tee -a ${currDir}/logs/output.log

 

…has the full absolute path to the scoop command's binary. The error message is telling you that the shell isn't finding the command, and your script is assuming that the binary will be in the path, which many times is not a valid assumption.

 

 

Bill Brooks, Community Manager
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Re: oozie shell action : sqoop command not found

Explorer

Do i need a bin folder of sqoop in hdfs?I do see that in local edge node in sqoop client but not in hdfs. I do see sqoop jar in hdfs.

Don't have an account?
Coming from Hortonworks? Activate your account here