Hi,
I've been trying to use the compress feature.
It does go through and shows succeeded. However, the file is found nowhere. Error below:
>>> Invoking Main class now >>>
Launcher class: class org.apache.oozie.action.hadoop.ShellMain
INFO: loading log4j config file log4j.properties.
INFO: log4j config file log4j.properties loaded successfully.
{"properties":[{"key":"oozie.launcher.job.id","value":"0001256-210527121043880-oozie-oozi-W","isFinal":false,"resource":"programmatically"},{"key":"oozie.job.id","value":"0001256-210527121043880-oozie-oozi-W","isFinal":false,"resource":"programmatically"},{"key":"oozie.action.id","value":"0001256-210527121043880-oozie-oozi-W@shell-a880","isFinal":false,"resource":"programmatically"},{"key":"mapreduce.job.tags","value":"oozie-2c0f7392133060a502e47be5e818a442","isFinal":false,"resource":"programmatically"}]}Setting [tez.application.tags] tag: oozie-2c0f7392133060a502e47be5e818a442
Setting [spark.yarn.tags] tag: oozie-2c0f7392133060a502e47be5e818a442
Current working dir /hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001
Copying /hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001/action.xml to /hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001/oozie-hadoop-conf-1638170749693/[core-site.xml, hdfs-site.xml, mapred-site.xml, yarn-site.xml]
Writing log4j.properties to /hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001/oozie-hadoop-conf-1638170749693
Setting HADOOP_CONF_DIR and YARN_CONF_DIR to /hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001/oozie-hadoop-conf-1638170749693
Full Command ..
-------------------------
0:compress_files_in_hdfs.sh:
1:-u=/user/hshrivastavahcl:
2:-f=Test New.txt:
3:-n=Test New.txt.zip:
List of passing environment
-------------------------
PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/kerberos/bin:
HADOOP_CONF_DIR=/hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001/oozie-hadoop-conf-1638170749693:
JAVA_HOME=/usr/java/jdk1.8.0_144-cloudera:
XFILESEARCHPATH=/usr/dt/app-defaults/%L/Dt:
APP_SUBMIT_TIME_ENV=1638170747786:
NM_HOST=ip-10-0-31-178.ec2.internal:
HADOOP_USER_NAME=hshrivastavahcl:
LD_LIBRARY_PATH=:
HADOOP_HDFS_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-hdfs:
LOGNAME=hshrivastavahcl:
JVM_PID=32267:
HADOOP_MAPRED_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-mapreduce:
PWD=/hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001:
HADOOP_COMMON_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop:
_=/usr/java/jdk1.8.0_144-cloudera/bin/java:
LOCAL_DIRS=/hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856:
APPLICATION_WEB_PROXY_BASE=/proxy/application_1636095248675_7856:
NM_HTTP_PORT=8042:
HADOOP_CLIENT_CONF_DIR=/etc/hadoop/conf.cloudera.yarn:
YARN_CONF_DIR=/hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001/oozie-hadoop-conf-1638170749693:
LOG_DIRS=/cdhlogs/yarn/container-logs/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001,/hadoop/yarn/container-logs/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001,/kafka/yarn/container-logs/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001:
PRELAUNCH_OUT=/kafka/yarn/container-logs/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001/prelaunch.out:
NM_AUX_SERVICE_mapreduce_shuffle=AAA0+gAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=:
NM_PORT=8041:
HADOOP_YARN_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-yarn:
USER=hshrivastavahcl:
CLASSPATH=/hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001:/hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001/*:/etc/hadoop/conf.cloudera.yarn:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-hdfs/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-yarn/*:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-yarn/lib/*::
PRELAUNCH_ERR=/kafka/yarn/container-logs/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001/prelaunch.err:
HADOOP_TOKEN_FILE_LOCATION=/hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001/container_tokens:
NM_AUX_SERVICE_spark_shuffle=:
NLSPATH=/usr/dt/lib/nls/msg/%L/%N.cat:
LOCAL_USER_DIRS=/hadoop/yarn/nm/usercache/hshrivastavahcl/:
OOZIE_ACTION_CONF_XML=/hadoop/yarn/nm/usercache/hshrivastavahcl/appcache/application_1636095248675_7856/container_e57_1636095248675_7856_01_000001/action.xml:
SHLVL=2:
HOME=/home/:
CONTAINER_ID=container_e57_1636095248675_7856_01_000001:
MALLOC_ARENA_MAX=4:
=================================================================
>>> Invoking Shell command line now >>
Stdoutput Created temporary output directory: /tmp/tmp.TOmCM835bH
Stdoutput Deleted temporary output directory: /tmp/tmp.TOmCM835bH
Exit code of the Shell command 1
<<< Invocation of Shell command completed <<<
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.oozie.action.hadoop.LauncherAM.runActionMain(LauncherAM.java:410)
at org.apache.oozie.action.hadoop.LauncherAM.access$300(LauncherAM.java:55)
at org.apache.oozie.action.hadoop.LauncherAM$2.run(LauncherAM.java:223)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.oozie.action.hadoop.LauncherAM.run(LauncherAM.java:217)
at org.apache.oozie.action.hadoop.LauncherAM$1.run(LauncherAM.java:153)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1875)
at org.apache.oozie.action.hadoop.LauncherAM.main(LauncherAM.java:141)
Caused by: org.apache.oozie.action.hadoop.LauncherMainException
at org.apache.oozie.action.hadoop.ShellMain.run(ShellMain.java:76)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:104)
at org.apache.oozie.action.hadoop.ShellMain.main(ShellMain.java:63)
... 16 more
Failing Oozie Launcher, Main Class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
Oozie Launcher, uploading action data to HDFS sequence file: hdfs://nameservice1/user/hshrivastavahcl/oozie-oozi/0001256-210527121043880-oozie-oozi-W/shell-a880--shell/action-data.seq
Stopping AM
Callback notification attempts left 0
Callback notification trying http://ip-10-0-21-22.ec2.internal:11000/oozie/callback?id=0001256-210527121043880-oozie-oozi-W@shell-a880&status=FAILED
Callback notification to http://ip-10-0-21-22.ec2.internal:11000/oozie/callback?id=0001256-210527121043880-oozie-oozi-W@shell-a880&status=FAILED succeeded
Callback notification succeeded
Am I missing something here?
Thanks,
Jawad
Announcements
Product Announcements
What's New @ Cloudera
Community Announcements
Community Announcements
Product Announcements
