Building Hive UDFs using HDP Specific Libraries

[#1]

I need to use the funnel Hive UDFs from here - https://github.com/yahoo/hive-funnel-udf

But I'm not sure if I should build the jar with HDP specific libraries? I was hoping if someone could help me with the details of what all needs to be changed in the dependency configuration for the build?

[#2 · Accepted solution]

You need to add repositories section to your maven pom file http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_user-guide/content/user-guide-setup-maven...

I've run many diffident applications with Apache versions of artifacts and you may be okay but as a good practice is advisable to match version of your dependencies to exact versions of your cluster.

[#3]

I see it's built for hadoop 1, you need to replace hadoop-core with hadoop-client. Hadoop version is 2.7.1 in Apache or 2.7.1.2.x.x.x in HDP. Again you might be okay with versions they provided except changing hadoop but having exact match is better.

[#4]

Additional info https://community.hortonworks.com/questions/1383/intelij-maven-hdp-public-repos-to-getting-indexed.h...

[#5]

Thanks Artem! I had to add 'http://repo.hortonworks.com/content/groups/public/' apart from the 'http://repo.hortonworks.com/content/repositories/releases/' repo to resolve for the following error I encountered -

[ERROR] Failed to execute goal on project funnel: Could not resolve dependencies for project com.yahoo.hive.udf:funnel:jar:1.0: The following artifacts could not be resolved: org.mortbay.jetty:jetty:jar:6.1.26.hwx, org.mortbay.jetty:jetty-util:jar:6.1.26.hwx: Failure to find org.mortbay.jetty:jetty:jar:6.1.26.hwx in http://repo.hortonworks.com/content/repositories/releases/ was cached in the local repository, resolution will not be reattempted until the update interval of HDP has elapsed or updates are forced -> [Help 1]

One another issue I face is that the code makes use of Java 8 features, while the sandbox Java version is "1.7.0_95". Should I consider upgrading Java in my Sandbox? or should I rewrite the code to make it compatible with Java 7?

[#6]

For jetty see this or do a search on our site, it's a common error https://community.hortonworks.com/questions/6256/jetty-util-6126hwxjar.html

You can install Java 8 side by side in sandbox

[#7]

Here's jetty fix https://community.hortonworks.com/questions/8663/unable-to-build-spark-application-due-to-missing-j....