Member since
10-16-2019
7
Posts
0
Kudos Received
0
Solutions
11-07-2019
05:40 PM
Can you please provide the answer how you were able to do it?
... View more
10-17-2019
08:19 PM
Do i need a bin folder of sqoop in hdfs?I do see that in local edge node in sqoop client but not in hdfs. I do see sqoop jar in hdfs.
... View more
10-16-2019
07:10 PM
I am running a shell script of sqoop through oozie which takes a list of tables/schema from list file in the script and load the data in data lake in kerberos cluster.
For running script we pass servername databasename as a argument.
It runs well in edge node but failing in oozie workflow with yarn logs as sqoop command not found.
Script sqoop_full.sh is given below:
#!/usr/bin/bash dmserver=$1 hdp_dir=$2
currDir=`realpath ${0} | rev | cut -d"/" -f2- | rev`
while read -r line do tbl=`echo $line | cut -d "|" -f1` dmdb=`echo $line | cut -d "|" -f2`
sqoop import -Dmapreduce.job.user.classpath.first=true -Dhadoop.security.credential.provider.path=jceks://hdfs/user/root/mssql.datamarthdpservice.pass.jceks \ --connect 'jdbc:jtds:sqlserver://'${dmserver}'.use.ucdp.net/'${dmdb}';integratedSecurity=true;domain=use.ucdp.net;database='${dmdb}';' \ --username datamhdpser \ --password-alias mssql.datamarthdpservice.pass \ --query "SELECT * from ${dmdb}.dbo.${tbl} WITH(NOLOCK) WHERE \$CONDITIONS" -m 1 \ --as-avrodatafile \ --compression-codec snappy \ --delete-target-dir \ --target-dir="hdfs:///data/collection/${hdp_dir}/${tbl}" 2>&1| tee -a ${currDir}/logs/output.log
done < <(tail -n +2 fulllist.txt | grep -v "#")
Workflow.xml is :
<?xml version="1.0" encoding="UTF-8" standalone="no"?> <workflow-app xmlns="uri:oozie:workflow:0.5" name="Edm_Sqoop_full"> <start to="shell_1"/> <action name="shell_1"> <shell xmlns="uri:oozie:shell-action:0.3"> <job-tracker>${resourceManager}</job-tracker> <name-node>${nameNode}</name-node> <configuration> <property> <name>oozie.use.system.libpath</name> <value>true</value> </property> <property> <name>oozie.libpath</name> <value>//user/oozie/share/lib/lib_20190213172012/sqoop</value> </property> <property> <name>hadoop.security.credential.provider.path</name> <value>jceks://hdfs/user/root/mssql.datamarthdpservice.pass.jceks</value> </property> <property> <name>mapreduce.user.classpath.first</name> <value>true</value> </property> <property> <name>mapreduce.task.classpath.user.precedence</name> <value>true</value> </property> </configuration> <exec>/user/sparkcronsvc/edm/sqoop_full.sh</exec> <argument>uso12smktdb01</argument> <argument>unica</argument> <file>/user/sparkcronsvc/edm/sqoop_full.sh</file> <file>/user/sparkcronsvc/edm/fulllist.txt</file> </shell> <ok to="end"/> <error to="kill"/> </action> <kill name="kill"> <message>${wf:errorMessage(wf:lastErrorNode())}</message> </kill> <end name="end"/> </workflow-app>
Yarn logs:
>>> Invoking Main class now >>>
Setting [tez.application.tags] tag: oozie-452d4388e731e788275adcc4b7f33c1d
Current working dir /data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002
Full Command ..
-------------------------
0:sqoop_full.sh:
1:uso12smktdb01:
2:unica:
List of passing environment
-------------------------
PATH=.:/usr/sbin:/sbin:/usr/lib/ambari-server/*:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/var/lib/ambari-agent:
HADOOP_CONF_DIR=/usr/hdp/2.6.4.125-1/hadoop/conf:
STDOUT_LOGFILE_ENV=/data/disk15/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/stdout:
HADOOP_ROOT_LOGGER=INFO,console:
JAVA_HOME=/usr/java/latest:
LANG=en_US.UTF-8:
NM_HOST=usor7dhdpw01.use.ucdp.net:
HADOOP_CLASSPATH=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/*:/data/disk8/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000001:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data/disk8/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000001/*:null:
LD_LIBRARY_PATH=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:/usr/hdp/2.6.4.125-1/hadoop/lib/native:/usr/hdp/2.6.4.125-1/hadoop/lib/native/Linux-amd64-64:./mr-framework/hadoop/lib/native:./mr-framework/hadoop/lib/native/Linux-amd64-64:
STDERR_LOGFILE_ENV=/data/disk15/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/stderr:
LOGNAME=sparkcronsvc:
JVM_PID=23287:
PWD=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:
HADOOP_CLIENT_OPTS=:
_=/usr/java/latest/bin/java:
LOCAL_DIRS=/data/disk1/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk2/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk3/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk4/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk5/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk6/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk7/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk8/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk9/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk10/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk11/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk12/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk13/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk14/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk15/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk16/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk18/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk19/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk20/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk21/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk22/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk23/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477,/data/disk24/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477:
SHELL=/bin/bash:
NM_HTTP_PORT=8042:
LOG_DIRS=/data/disk1/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk2/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk3/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk4/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk5/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk6/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk7/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk8/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk9/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk10/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk11/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk12/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk13/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk14/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk15/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk16/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk17/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk18/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk19/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk20/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk21/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk22/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk23/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002,/data/disk24/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:
PRELAUNCH_OUT=/data/disk1/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/prelaunch.out:
NM_AUX_SERVICE_mapreduce_shuffle=AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
:
NM_PORT=45454:
HADOOP_YARN_HOME=/usr/hdp/2.6.4.125-1/hadoop-yarn:
USER=sparkcronsvc:
CLASSPATH=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/mapreduce/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/common/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/common/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/yarn/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/yarn/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/hdfs/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/mr-framework/hadoop/share/hadoop/tools/lib/*:/usr/hdp/2.6.4.125-1/hadoop/lib/hadoop-lzo-0.6.0.2.6.4.125-1.jar:/etc/hadoop/conf/secure:/usr/hdp/current/ext/hadoop/*:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/*:
PRELAUNCH_ERR=/data/disk1/hadoop/yarn/log/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/prelaunch.err:
HADOOP_TOKEN_FILE_LOCATION=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/container_tokens:
NM_AUX_SERVICE_spark_shuffle=:
LOCAL_USER_DIRS=/data/disk1/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk2/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk3/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk4/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk5/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk6/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk7/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk8/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk9/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk10/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk11/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk12/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk13/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk14/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk15/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk16/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk18/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk19/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk20/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk21/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk22/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk23/hadoop/yarn/local/usercache/sparkcronsvc/,/data/disk24/hadoop/yarn/local/usercache/sparkcronsvc/:
HADOOP_HOME=/usr/hdp/2.6.4.125-1/hadoop:
OOZIE_ACTION_CONF_XML=/data/disk17/hadoop/yarn/local/usercache/sparkcronsvc/appcache/application_1559931840942_7477/container_e45_1559931840942_7477_01_000002/action.xml:
SHLVL=1:
HOME=/home/:
NM_AUX_SERVICE_spark2_shuffle=:
CONTAINER_ID=container_e45_1559931840942_7477_01_000002:
MALLOC_ARENA_MAX=4:
=================================================================
>>> Invoking Shell command line now >>
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
Stdoutput ./sqoop_full.sh: line 44: sqoop: command not found
... View more
Labels:
- Labels:
-
Apache Oozie
-
Apache Sqoop