Reply
Highlighted
Contributor
Posts: 28
Registered: ‎09-20-2017
Accepted Solution

hadoop-core and hive-exec versions for UDF

Hello,

 

How should I choose hadoop-core and hive-exec jar versions for Impala UDFs? What should they match in the target CDH installation?

 

For instance, hive-exec is now at version 3.1.0 but most examples I've seen use 0.13.x...

 

Thank you,

Gerasimos

Cloudera Employee
Posts: 368
Registered: ‎07-29-2015

Re: hadoop-core and hive-exec versions for UDF

I'd recommend using the versions matching your CDH installation.
Contributor
Posts: 28
Registered: ‎09-20-2017

Re: hadoop-core and hive-exec versions for UDF

That is what I am asking. I have CDH 5.11, which reports:

 

hadoop-2.6.0+cdh5.11.2+2429

- hive-1.1.0+cdh5.11.2+1082

 

Does this mean that I have to use hadoop-core-2.6.0 ? There isn't such version, as the max is hadoop-core-1.2.1.

 

Cloudera Employee
Posts: 368
Registered: ‎07-29-2015

Re: hadoop-core and hive-exec versions for UDF

We have a CDH Maven repository with the CDH version of things. CDH releases don't always align exactly with Apache releases - they may include additional bug fixes, features or may have non-production-ready features disabled.

 

I believe these are the docs for how to reference them via maven:

https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_vd_cdh5_maven_repo.html

 

E.g. here is hadoop-core for 5.11.2: https://repository.cloudera.com/artifactory/cloudera-repos/org/apache/hadoop/hadoop-core/2.6.0-mr1-c...

Contributor
Posts: 28
Registered: ‎09-20-2017

Re: hadoop-core and hive-exec versions for UDF

[ Edited ]

Thank you,

 

I added cloudera repo to my gradle configuration and builds and functions run from Impala without problems.

Announcements