how to add third party jars for a mapreduce program

[#1]

Hi All,

Am getting this error

cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.vertica.hadoop.VerticaOutputFormat not found
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.vertica.hadoop.VerticaOutputFormat not found

However , I have added the third party jar into the /usr/hdp/2.3.4.0-***/hadoop/lib

and have also used -libjar option while running the command .

can anyone let me know where to set the hadoop classpath correctly , and if we are copying the third party jars into the hadoop source folder there are many folder within like hadoop-hdfs,hadoop-mapreduce,hadoop-httpf,hadoop-yarn do we copy the third party jars here too ????

[#2 · Accepted solution]

You need to add jars with -libjars not -libjar

you can also add your connector jar as dependency to your project as I am not able to find a maven pom you can include. You also need to make sure if you do put the jar in lib of hadoop ditectories, you need to place it on every node.

You can also get familiar with DistributedCache API to upload jars in hdfs for access. https://hadoop.apache.org/docs/current/api/org/apache/hadoop/filecache/DistributedCache.html