Reply
New Contributor
Posts: 2
Registered: ‎10-09-2014

Unable to write to HDFS from Oozie shell action

I am invoking the below Oozie shell action to invoke a shell script which writes an output to a file. I have tried this on Cloudera 5.7 VM ...Whenever I am trying to write to a new file in HDFS the job fails.

I checked the application/job id logs but had no info. Is there any restriction to write to hdfs from a shell script?

 

In the shell script below, I am trying to query on a table and write the results to a file.

 

---sample.sh---

hive -e "create t1(id int); insert into t1 values(1);"
hive -e "select * from t1;" > /user/cloudera/test123/t1_data.txt


--workflow.xml

<?xml version="1.0" encoding="UTF-8"?>

<workflow-app xmlns="uri:oozie:workflow:0.5" name="shell.workflow">
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<prepare>
<delete path="test123"/>
<mkdir path="test123"/>
</prepare>

<configuration>
<property>
<name>mapred.job.queue.name</name>
<value>${queueName}</value>
</property>
</configuration>
<exec>sample.sh</exec>
<env-var>HADOOP_USER_NAME=cloudera</env-var>
<file>hdfs://localhost:8020/user/cloudera/oozietest/sample.sh#sample.sh</file>
<capture-output/>
</shell>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Error in Shell.Please refer the Oozie Logs</message>
</kill>
<end name="end"/>
</workflow-app>


--job.properties

nameNode=hdfs://localhost:8020
jobTracker=localhost:8032
master=yarn-cluster
user.name=cloudera
queueName=default
examplesRoot=oozietest
oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}

 

 

Posts: 177
Topics: 8
Kudos: 28
Solutions: 19
Registered: ‎07-16-2015

Re: Unable to write to HDFS from Oozie shell action

Hi,

 

First of all, the following command will not write to HDFS but to the local FS of the node executing the task.

"hive -e "select * from t1;" > /user/cloudera/test123/t1_data.txt"

 

Does this path exist ? And does the user runing the shell have write permission on that path (on the local FS).

Announcements