How to choose hive-exec jar version for Custom UDF

[#1]

hi Guys,

Could someone let me know how they decide which hive-exec jar version would be compatible with their Environment.

Here are the two approches I have take, the first one is failing and second one passes. However second approach is really messy and I would like to have first approach work somehow.

First Approach - Not Working

• use the maven project to compile and build single (fat) jar.
• check which version of hive-exec.jar hadoop is using from following directory

	/usr/hdp/current/hive-client/lib/hive-exec.jar -> hive-exec-1.2.1000.2.6.4.0-91.jar

• use the matching version as maven dependancy

<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-exec</artifactId>
    <version>1.2.1</version>
</dependency>

Second Approach - Working

• Create the simple JAVA project (not the maven)
• copy the JAR from hadoop and add in classpath
• compile and create the class files
• Create a thin Jar (without any dependancies), Provide class path in mainfest.mf file

Class-path: /usr/hdp/current/hive-client/lib/hive-exec.jar

[#2]

Use hortonworks public repo & try with below maven dependency to compile custom UDF with specific HDP version.

<dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-exec</artifactId>
    <version>1.2.1000.2.6.4.0-91</version>
</dependency>

Hortonworks maven public repo can be referred from this link

[#3]

@Naresh P R

Thanks, Now I can resolve the dependancy However, not able to compile the code. I am getting following error.

I tried to add the dependancy mentioned, However it's not helping either.

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-assembly-plugin:2.2-beta-5:single (default-cli) on project hive-normalize: Failed to create assembly: Failed to resolve dependencies for project: ca.abc:hive-normalize:jar:1.0: Missing:
[ERROR] ----------
[ERROR] 1) org.mortbay.jetty:jetty-util:jar:6.1.ca26.hwx
[ERROR] 
[ERROR] Try downloading the file manually from the project website.
[ERROR] 
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=org.mortbay.jetty -DartifactId=jetty-util -Dversion=6.1.26.hwx -Dpackaging=jar -Dfile=/path/to/file
[ERROR] 
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=org.mortbay.jetty -DartifactId=jetty-util -Dversion=6.1.26.hwx -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR] 
[ERROR] Path to dependency:
[ERROR] 1) ca.abc:hive-normalize:jar:1.0
[ERROR] 2) org.apache.hive:hive-exec:jar:1.2.1000.2.6.4.0-91
[ERROR] 3) org.apache.hive:hive-shims:jar:1.2.1000.2.6.4.0-91
[ERROR] 4) org.apache.hive.shims:hive-shims-0.23:jar:1.2.1000.2.6.4.0-91
[ERROR] 5) org.apache.hadoop:hadoop-hdfs:jar:2.7.3.2.6.4.0-91
[ERROR] 6) org.mortbay.jetty:jetty-util:jar:6.1.26.hwx
[ERROR] 
[ERROR] 2) org.mortbay.jetty:jetty:jar:6.1.26.hwx
[ERROR] 
[ERROR] Try downloading the file manually from the project website.
[ERROR] 
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=org.mortbay.jetty -DartifactId=jetty -Dversion=6.1.26.hwx -Dpackaging=jar -Dfile=/path/to/file
[ERROR] 
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=org.mortbay.jetty -DartifactId=jetty -Dversion=6.1.26.hwx -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR] 
[ERROR] Path to dependency:
[ERROR] 1) ca.abc:hive-normalize:jar:1.0
[ERROR] 2) org.apache.hive:hive-exec:jar:1.2.1000.2.6.4.0-91
[ERROR] 3) org.apache.hive:hive-shims:jar:1.2.1000.2.6.4.0-91
[ERROR] 4) org.apache.hive.shims:hive-shims-0.23:jar:1.2.1000.2.6.4.0-91
[ERROR] 5) org.apache.hadoop:hadoop-hdfs:jar:2.7.3.2.6.4.0-91
[ERROR] 6) org.mortbay.jetty:jetty:jar:6.1.26.hwx
[ERROR] 
[ERROR] 3) org.mortbay.jetty:jetty-sslengine:jar:6.1.26.hwx
[ERROR] 
[ERROR] Try downloading the file manually from the project website.
[ERROR] 
[ERROR] Then, install it using the command:
[ERROR] mvn install:install-file -DgroupId=org.mortbay.jetty -DartifactId=jetty-sslengine -Dversion=6.1.26.hwx -Dpackaging=jar -Dfile=/path/to/file
[ERROR] 
[ERROR] Alternatively, if you host your own repository you can deploy the file there:
[ERROR] mvn deploy:deploy-file -DgroupId=org.mortbay.jetty -DartifactId=jetty-sslengine -Dversion=6.1.26.hwx -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[ERROR] 
[ERROR] Path to dependency:
[ERROR] 1) ca.abc:hive-normalize:jar:1.0
[ERROR] 2) org.apache.hive:hive-exec:jar:1.2.1000.2.6.4.0-91
[ERROR] 3) org.apache.hive:hive-shims:jar:1.2.1000.2.6.4.0-91
[ERROR] 4) org.apache.hive.shims:hive-shims-0.23:jar:1.2.1000.2.6.4.0-91
[ERROR] 5) org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:2.7.3.2.6.4.0-91
[ERROR] 6) org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.3.2.6.4.0-91
[ERROR] 7) org.apache.hadoop:hadoop-yarn-registry:jar:2.7.3.2.6.4.0-91
[ERROR] 😎 org.apache.hadoop:hadoop-common:jar:2.7.3.2.6.4.0-91
[ERROR] 9) org.mortbay.jetty:jetty-sslengine:jar:6.1.26.hwx
[ERROR] 
[ERROR] ----------
[ERROR] 3 required artifacts are missing.
[ERROR] 
[ERROR] for artifact:
[ERROR] ca.abc:hive-normalize:jar:1.0
[ERROR] 
[ERROR] from the specified remote repositories:
[ERROR] hortonworks.extrepo (http://repo.hortonworks.com/content/repositories/releases, releases=true, snapshots=true),
[ERROR] central (https://repo.maven.apache.org/maven2, releases=true, snapshots=false)
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

[#4]

@Gaurang Shah

You are using hortonworks.extrepo (http://repo.hortonworks.com/content/repositories/releases, releases=true, snapshots=true),

This repo dont have dependent jars, can you please use http://nexus-private.hortonworks.com/nexus/content/groups/public/

This repo has all the required dependencies

eg., 1) org.mortbay.jetty:jetty-util:jar:6.1.ca26.hwx

http://nexus-private.hortonworks.com/nexus/content/groups/public/org/mortbay/jetty/jetty-util/6.1.26...

[#5]

@Naresh P R I am using following repo and it resolve my issue

http://repo.hortonworks.com/content/groups/public/

[#6]

I am glad the problem is resolved.

Both

1) http://nexus-private.hortonworks.com/nexus/content/groups/public/

2) http://repo.hortonworks.com/content/groups/public/

Will resolve this issue as mentioned in my previous link to refer hortonworks public repo. Please accept the answer