Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Oozie fails when using map reduce command inside pig script

Oozie fails when using map reduce command inside pig script

New Contributor

Hi,

I am using a Mapreduce command inside pig script.

RULEENG_DATA = MAPREDUCE 'ruleeng.jar' STORE FILTERED_DATA INTO '/data/data-analytics/temp/mroutput/inputDir' LOAD '/data/data-analytics/temp/mroutput/outputDir' AS (key:chararray,date:chararray,date_time:chararray,year_month:chararray,year:chararray,hit_time_gmt:chararray,post_visid_high:chararray,post_visid_low:chararray,ip:chararray) `com.stubhub.mapreduce.ruleengine.RuleEngineDriver /data/data-analytics/temp/mroutput/inputDir /data/data-analytics/temp/mroutput/outputDir 23`;

it is throwing an error when i run the pig action through oozie

***log file***

Caused by: java.lang.ClassNotFoundException: com.stubhub.mapreduce.ruleengine.RuleEngineDriver
	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
	at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.NativeMapReduceOper.runJob(NativeMapReduceOper.java:80)

***workflow.xml***

<script>fact_traffic.pig</script> <argument>-param_file</argument> <argument>param_job_fact_trafficconfig_fact_traffic.txt</argument> <file>lib/hive-site.xml</file> <file>hdfs://hdfs-cluster//user/bdappuser/workflows/traffic-ft-wf/lib/piggybank.jar#piggybank.jar</file> <file>hdfs://hdfs-cluster//user/bdappuser/workflows/traffic-ft-wf/lib/pigloaderhbasefin.jar#pigloaderhbasefin.jar</file> <file>hdfs://hdfs-cluster//user/bdappuser/workflows/traffic-ft-wf/lib/ruleeng.jar#ruleeng.jar</file> <file>hdfs://hdfs-cluster//user/bdappuser/workflows/traffic-ft-wf/lib/hadoop-mapreduce-examples-2.7.1.2.3.4.0-3485.jar#hadoop-mapreduce-examples-2.7.1.2.3.4.0-3485.jar</file>

2 REPLIES 2
Highlighted

Re: Oozie fails when using map reduce command inside pig script

Rising Star
@Navdeep Parmar

Have you included jar for RuleEngineDriver class in your workflow lib? Seems you are using third party dependency which are not included in default classpath or oozie shared lib. Include the required dependency either in workflow lib or place them in oozie shared lib.

Re: Oozie fails when using map reduce command inside pig script

New Contributor

Yes RuleEngineDriver is the main class of ruleeng.jar and is included in the workflows/traffic-ft-wf/lib/ruleeng.jar.. This is a custom java code. should this be part of the oozie shared lib path?

Don't have an account?
Coming from Hortonworks? Activate your account here