Support Questions
Find answers, ask questions, and share your expertise

Cannot find log file in app-logs directory while running Pig action from Oozie workflow

Cannot find log file in app-logs directory while running Pig action from Oozie workflow

Explorer

I have created Oozie workflow for Hive and Pig action on Azure hortonworks cloudera But while running Oozie workflow Hive action run successfully but Pig action throws below error :

 

19/12/10 14:11:59 INFO client.RMProxy: Connecting to ResourceManager at bigdata-m0.xxxxxxx.cloudapp.net/10.0.0.7:8050
19/12/10 14:12:00 INFO client.AHSProxy: Connecting to Application History server at bigdata-m0.xxxx.cloudapp.net/10.0.0.7:10200
File /app-logs/hdfs/logs/application_15770947956_0159 does not exist.
Can not find any log file matching the pattern: [ALL] for the application: application_15770947956_0159
Can not find the logs for the application: application_15770947956_0159 with the appOwner: hdfs

 

 

When I run the pig script using 'pig -x tez -useHCatalog -f script.pig' then it run successfully but while running from Oozie workflow it throws above error so I added shell action before Pig action to give permission to the directory [/app-logs/hdfs/logs/] still it throws above error.

workflow.xml :

 

<workflow-app xmlns="uri:oozie:workflow:0.5" name="PigHiveworkflow">
    <start to="hive-node"/>
    <action name="hive-node">
        <hive2 xmlns="uri:oozie:hive2-action:0.2">
            <job-tracker>${resourceManager}</job-tracker>
            <name-node>${nameNode}</name-node>
            <jdbc-url>jdbc:hive2://xxxxxxxxxxx:2181/default;httpPath=cliservice;serviceDiscoveryMode=zooKeeper;transportMode=http;user=xxxx;zooKeeperNamespace=hiveserver2</jdbc-url>
            <password>xxxx</password>
            <script>${appPath}/hive.hql</script>
        </hive2>
        <ok to="shell-node"/>
        <error to="kill"/>
    </action>

  <action name="shell-node">
        <shell xmlns="uri:oozie:shell-action:0.3">
            <job-tracker>${resourceManager}</job-tracker>
            <name-node>${nameNode}</name-node>
                       <exec>script.sh</exec>           
            <file>${appPath}/script.sh</file>
            <capture-output/>
        </shell>
        <ok to="pig-node"/>
        <error to="kill"/>
    </action>

<action name="pig-node">
            <pig>
            <job-tracker>${resourceManager}</job-tracker>
            <name-node>${nameNode}</name-node>
            <job-xml>${appPath}/hive-site.xml</job-xml>
            <configuration>
                <property>
                    <name>oozie.action.sharelib.for.pig</name>
                    <value>pig,hive,hcatalog</value>
                </property>
            </configuration>
            <script>${appPath}/demo.pig</script>
        </pig>
    <ok to="end"/>
    <error to="kill"/>
  </action>
  <kill name="kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
  </kill>
  <end name="end"/>
</workflow-app>

 

I tried to give permission to directory [/app-logs/hdfs/logs] using fs command in Pig script:

fs -chmod -R 777 /app-logs/root;

Still Pig action in Oozie workflow throws error.

I am running Oozie workflow as a hdfs user.I am not getting which file permission is still missing.

 

Thanks,