Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

HIve UDF oozie

HIve UDF oozie

Explorer

I have followed the below link to configure hive udf :-

 

https://www.cloudera.com/documentation/enterprise/5-6-x/topics/cm_mc_hive_udf.html

 

When I run my query with Hive Editor in Hue.It recognises the UDF but when running the same query with hive action  oozie it gives the error :-

 

 

4924 [main] ERROR org.apache.hadoop.hive.ql.Driver  - FAILED: SemanticException Failed to breakup Windowing invocations into Groups. At least 1 group must only depend on input columns. Also check for circular dependencies.
Underlying error: org.apache.hadoop.hive.ql.parse.SemanticException: Line 26:44 Invalid function 'UDF_DateFormat'
org.apache.hadoop.hive.ql.parse.SemanticException: Failed to breakup Windowing invocations into Groups. At least 1 group must only depend on input columns. Also check for circular dependencies.
Underlying error: org.apache.hadoop.hive.ql.parse.SemanticException: Line 26:44 Invalid function 'UDF_DateFormat'
	at org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:94)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:12051)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8820)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8778)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9645)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9525)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9552)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9538)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10011)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10022)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9902)
	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:446)
	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:312)
	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1201)
	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1296)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1127)
	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1115)
	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:220)
	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:383)
	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:318)
	at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:416)
	at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:432)
	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:726)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:693)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:628)
	at org.apache.oozie.action.hadoop.HiveMain.runHive(HiveMain.java:325)
	at org.apache.oozie.action.hadoop.HiveMain.run(HiveMain.java:302)
	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:49)
	at org.apache.oozie.action.hadoop.HiveMain.main(HiveMain.java:69)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)

and before this in stdout logs it says:-

3132 [main] INFO  hive.ql.metadata.Hive  - Registering function udf_dateformat com.org.tes.DateFormat

 

2 REPLIES 2

Re: HIve UDF oozie

Cloudera Employee

Are you using a Hive or a Hive2 action? The description you've linked talks about Hive2.

The stack trace looks more like a Hive action. 

 

Also, if switching to Hive2 does not help, you may try to add the UDF jar into the lib folder near your wokrflow.xml

Re: HIve UDF oozie

Explorer

I am using Hive2.

I resolved it by creating temporary function per session of Hive query.