Reply
Highlighted
Explorer
Posts: 13
Registered: ‎04-01-2016

How to pass -libjars parameter to yarn ?

quickstart vm, chd 5.8

1) I've copied avro-mapred-1.8.0.jar to /usr/lib/hadoop/lib, my job is starting OK.

2) mv /usr/lib/hadoop/lib/avro-mapred-1.8.0.jar /home/cloudera/

 

# yarn jar target/testmr-1.0-SNAPSHOT.jar ParquetJob -libjars /home/cloudera/avro-mapred-1.8.0.jar /user/hive/warehouse/xy_parquet/3a45bcbc3f1b744f-33b7752baf856aa6_1977916394_data.0.parq out_file1
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/parquet-hadoop-bundle-1.5.0-cdh5.8.0.jar!/shaded/parquet/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/parquet-format-2.1.0-cdh5.8.0.jar!/shaded/parquet/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/parquet/lib/parquet-pig-bundle-1.5.0-cdh5.8.0.jar!/shaded/parquet/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [shaded.parquet.org.slf4j.helpers.NOPLoggerFactory]
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/avro/mapred/AvroValue
at ParquetJob.run(ParquetJob.java:61)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at ParquetJob.main(ParquetJob.java:28)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: java.lang.ClassNotFoundException: org.apache.avro.mapred.AvroValue
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 9 more

Announcements