Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Regarding oozie workflow with shell command and apache falcon

Regarding oozie workflow with shell command and apache falcon

New Contributor

Hello I am running apache falcon and oozie with shell script to run the workflow.

 

getting this error

ell-node] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
2014-11-26 17:24:06,645  INFO ActionEndXCommand:543 - SERVER[sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000020-141126162609456-oozie-oozi-W] ACTION[0000020-141126162609456-oozie-oozi-W@shell-node] ERROR is considered as FAILED for SLA
2014-11-26 17:24:06,702  INFO ActionStartXCommand:543 - SERVER[sandbox.hortonworks.com] USER[ambari-qa] GROUP[-] TOKEN[] APP[shell-wf] JOB[0000020-141126162609456-oozie-oozi-W] ACTION[0000020-141126162609456-oozie-oozi-W@fail] Start action [0000020-141126162609456-oozie-

 

 

Workflow

<!--
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
  regarding copyright ownership.  The ASF licenses this file
  to you under the Apache License, Version 2.0 (the
  "License"); you may not use this file except in compliance
  with the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

  Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License.
-->
<workflow-app xmlns="uri:oozie:workflow:0.4" name="shell-wf">
    <start to="shell-node"/>
    <action name="shell-node">
        <shell xmlns="uri:oozie:shell-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <configuration>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <exec>ingest.sh</exec>
            <argument>${feedInstancePaths}</argument>
            <file>${wf:appPath()}/ingest.sh#ingest.sh</file>
            <!-- <file>/tmp/ingest.sh#ingest.sh</file> -->
            <!-- <capture-output/> -->
        </shell>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Shell action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

 

 

 

This is the shell script

 

#!/bin/bash

java -cp "/media/sf_ToSandbox/BigData-0.0.1-SNAPSHOT.jar:/usr/hdp/2.2.0.0-913/sqoop/lib/*:/usr/hdp/2.2.0.0-913/hadoop/lib/*" com.bigdata.falcon.StaticDataProvider

 

These are the feed and process and cluster information

 

<feed description="Raw customer email feed" name="FINALrawEmailFeed"
    xmlns="uri:falcon:feed:0.1">

    <tags>externalSystem=USWestEmailServers,classification=secure</tags>
    <groups>churnAnalysisDataPipeline</groups>
    <frequency>minutes(3)</frequency>
    <late-arrival cut-off="hours(4)"/>
    <clusters>
        <cluster name="SprimaryCluster" type="source">
            <validity start="2014-11-21T00:00Z" end="2016-03-31T00:00Z"/>
            <retention limit="days(90)" action="delete"/>
        </cluster>
    </clusters>

    <locations>
        <location type="data"
        path="/user/gaurav123/falcon/demo/primary/input/enron/${YEAR}-${MONTH}-${DAY}-${HOUR}"/>
        <location type="stats" path="/none"/>
        <location type="meta" path="/none"/>
    </locations>

    <ACL owner="ambari-qa" group="users" permission="0777"/>
    <schema location="/none" provider="none"/>
</feed>

 

Process

-----------

 

<?xml version="1.0" encoding="UTF-8"?>
<!--
 Hourly process, ingests raw emails from all US west email servers
 -->
<process name="IngestProcess" xmlns="uri:falcon:process:0.1">

    <tags>pipeline=churnAnalysisDataPipeline,owner=ETLGroup,externalSystem=USWestEmailServers</tags>


    <clusters>
        <cluster name="SprimaryCluster">
            <validity start="2014-11-24T09:35Z" end="2014-11-25T10:00Z"/>
        </cluster>
    </clusters>

    <parallel>1</parallel>
    <order>FIFO</order>
    <frequency>minutes(3)</frequency>

    <outputs>
        <output name="output" feed="FINALrawEmailFeed" instance="now(0,0)" />
    </outputs>

    <workflow name="IngestWorkflow" version="2.0.0"
    engine="oozie" path="/user/gaurav123/falcon/demo/apps/ingest/fs" />

    <retry policy="periodic" delay="minutes(15)" attempts="3" />

</process>

 

 

Can you please let me know what is the issue , I have tried since long two days? but unable to run the workflow with falcon process

 

 

 

 

3 REPLIES 3

Re: Regarding oozie workflow with shell command and apache falcon

New Contributor

Also got this exception

 

2014-11-26 17:47:42,548 ERROR CompletedActionXCommand:540 - SERVER[sandbox.hortonworks.com] USER[-] GROUP[-] TOKEN[] APP[-] JOB[0000099-141126162609456-oozie-oozi-W] ACTION[0000099-141126162609456-oozie-oozi-W@user-action] XException, 
org.apache.oozie.command.CommandException: E0800: Action it is not running its in [PREP] state, action [0000099-141126162609456-oozie-oozi-W@user-action]

Re: Regarding oozie workflow with shell command and apache falcon

Master Guru
An action being unable to transition beyond PREP state indicates that the Oozie server was unable to complete pre-requisites in order to submit it. I'd recommend taking a look at the Oozie log, looking for your Oozie Job ID's most earlier entries in it (0000099-141126162609456-oozie-oozi-W), to find the actual error that is preventing its submission.

Oozie job failing

New Contributor

132792-oozie-oozi-W] ACTION[0000002-170813044132792-oozie-oozi-W@python-node] Launcher exception: Output data exceeds its limit [8192]. 

 

the input is a txt file with just a few lines. Overall file size is 1kb