Support Questions

Find answers, ask questions, and share your expertise

Building Hive UDFs using HDP Specific Libraries

avatar

I need to use the funnel Hive UDFs from here - https://github.com/yahoo/hive-funnel-udf

But I'm not sure if I should build the jar with HDP specific libraries? I was hoping if someone could help me with the details of what all needs to be changed in the dependency configuration for the build?

1 ACCEPTED SOLUTION

avatar
Master Mentor

You need to add repositories section to your maven pom file http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_user-guide/content/user-guide-setup-maven...

I've run many diffident applications with Apache versions of artifacts and you may be okay but as a good practice is advisable to match version of your dependencies to exact versions of your cluster.

View solution in original post

6 REPLIES 6

avatar
Master Mentor

You need to add repositories section to your maven pom file http://docs.hortonworks.com/HDPDocuments/HDP1/HDP-1.3.1/bk_user-guide/content/user-guide-setup-maven...

I've run many diffident applications with Apache versions of artifacts and you may be okay but as a good practice is advisable to match version of your dependencies to exact versions of your cluster.

avatar
Master Mentor

I see it's built for hadoop 1, you need to replace hadoop-core with hadoop-client. Hadoop version is 2.7.1 in Apache or 2.7.1.2.x.x.x in HDP. Again you might be okay with versions they provided except changing hadoop but having exact match is better.

avatar
Master Mentor

avatar

Thanks Artem! I had to add 'http://repo.hortonworks.com/content/groups/public/' apart from the 'http://repo.hortonworks.com/content/repositories/releases/' repo to resolve for the following error I encountered -

[ERROR] Failed to execute goal on project funnel: Could not resolve dependencies for project com.yahoo.hive.udf:funnel:jar:1.0: The following artifacts could not be resolved: org.mortbay.jetty:jetty:jar:6.1.26.hwx, org.mortbay.jetty:jetty-util:jar:6.1.26.hwx: Failure to find org.mortbay.jetty:jetty:jar:6.1.26.hwx in http://repo.hortonworks.com/content/repositories/releases/ was cached in the local repository, resolution will not be reattempted until the update interval of HDP has elapsed or updates are forced -> [Help 1]

One another issue I face is that the code makes use of Java 8 features, while the sandbox Java version is "1.7.0_95". Should I consider upgrading Java in my Sandbox? or should I rewrite the code to make it compatible with Java 7?

avatar
Master Mentor

For jetty see this or do a search on our site, it's a common error https://community.hortonworks.com/questions/6256/jetty-util-6126hwxjar.html

You can install Java 8 side by side in sandbox

avatar
Master Mentor