Support Questions
Find answers, ask questions, and share your expertise

how to add third party jars for a mapreduce program

Explorer

Hi All,

Am getting this error

cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.vertica.hadoop.VerticaOutputFormat not found
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.vertica.hadoop.VerticaOutputFormat not found

However , I have added the third party jar into the /usr/hdp/2.3.4.0-***/hadoop/lib

and have also used -libjar option while running the command .

can anyone let me know where to set the hadoop classpath correctly , and if we are copying the third party jars into the hadoop source folder there are many folder within like hadoop-hdfs,hadoop-mapreduce,hadoop-httpf,hadoop-yarn do we copy the third party jars here too ????

1 ACCEPTED SOLUTION

Accepted Solutions

Mentor

You need to add jars with -libjars not -libjar

you can also add your connector jar as dependency to your project as I am not able to find a maven pom you can include. You also need to make sure if you do put the jar in lib of hadoop ditectories, you need to place it on every node.

You can also get familiar with DistributedCache API to upload jars in hdfs for access. https://hadoop.apache.org/docs/current/api/org/apache/hadoop/filecache/DistributedCache.html

View solution in original post

1 REPLY 1

Mentor

You need to add jars with -libjars not -libjar

you can also add your connector jar as dependency to your project as I am not able to find a maven pom you can include. You also need to make sure if you do put the jar in lib of hadoop ditectories, you need to place it on every node.

You can also get familiar with DistributedCache API to upload jars in hdfs for access. https://hadoop.apache.org/docs/current/api/org/apache/hadoop/filecache/DistributedCache.html

View solution in original post