Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

3rd jackson-core-*.jar on HDP install

avatar
Master Guru

I am finding many libraries part of HDP base install are added to class path like jackson-core-2.2.3.jar. However these libraries do not come with the vanilla (non hdp) install. Does anyone know why and how these libraries are used? If new version of the library exist, HDP may force to specify classpath for each application running with different jar. Is there a possible work around where HDP separate core hadoop libraries classpth from non core libraries (like jackson..*)?

1 ACCEPTED SOLUTION

avatar
Master Guru

There is no solution for this at the moment. I have informed @Tim Hall. Work around is using maven shading.

View solution in original post

7 REPLIES 7

avatar
Master Mentor
@Sunile Manjee

Interesting question 🙂

Core Jackson abstractions, basic JSON streaming API implementation

http://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-core/2.2.3

avatar
Master Guru

There is no solution for this at the moment. I have informed @Tim Hall. Work around is using maven shading.

avatar
Explorer

The presence of this dependency is causing issues for users using both HDP and CDH it seems; https://github.com/druid-io/druid/blob/master/docs/content/operations/other-hadoop.md#cdh

If its not a necessary component and the work to extract it is relatively small it would save a lot of headache if it can be extracted from core libraries

avatar
Master Mentor

Use maven exclude for this jar as a workaround when you deploy jars.

avatar
Explorer

@Artem Ervits Care to elaborate? Which jar are you talking about? when we deploy Hadoop, HDP, Druid or our Job? Excluding the dependency in Druid is the headache since the versions conflict. if I understood this question correctly @Sunile Manjee suggests separating the Jackson dependencies from the core Hadoop libraries which should solve this and future version conflicts for Jackson

avatar
Master Mentor

You can either parse your classpatg and remove the dependency programmatically or if you are getting an error when you deploy an application, use <exclude> tag ofaven to remove a conflicting, usually an older version of Jackson from your build. In my case it was the latter, maven was pulling an older version of Jackson core and in classpatg it was appearing before the one required by my mapreduce program.

avatar
New Contributor

is this fixed? We are facing same issue with json-smart.jar.. My application uses 2.1 version, whereas hadoop 2.6 has 1.1.1 version in the classpath..