Support Questions
Find answers, ask questions, and share your expertise

Unable to write to HDFS from Oozie shell action

Unable to write to HDFS from Oozie shell action

New Contributor

I am invoking the below Oozie shell action to invoke a shell script which writes an output to a file. I have tried this on Cloudera 5.7 VM ...Whenever I am trying to write to a new file in HDFS the job fails.

I checked the application/job id logs but had no info. Is there any restriction to write to hdfs from a shell script?


In the shell script below, I am trying to query on a table and write the results to a file.

hive -e "create t1(id int); insert into t1 values(1);"
hive -e "select * from t1;" > /user/cloudera/test123/t1_data.txt


<?xml version="1.0" encoding="UTF-8"?>

<workflow-app xmlns="uri:oozie:workflow:0.5" name="shell.workflow">
<start to="shell-node"/>
<action name="shell-node">
<shell xmlns="uri:oozie:shell-action:0.1">
<delete path="test123"/>
<mkdir path="test123"/>

<ok to="end"/>
<error to="kill"/>
<kill name="kill">
<message>Error in Shell.Please refer the Oozie Logs</message>
<end name="end"/>





Re: Unable to write to HDFS from Oozie shell action

Super Collaborator



First of all, the following command will not write to HDFS but to the local FS of the node executing the task.

"hive -e "select * from t1;" > /user/cloudera/test123/t1_data.txt"


Does this path exist ? And does the user runing the shell have write permission on that path (on the local FS).