Member since
10-10-2016
3
Posts
7
Kudos Received
0
Solutions
10-10-2016
11:05 AM
Thank you very much, with -Dmapreduce.job.user.classpath.first=true works like a charm!
... View more
10-10-2016
08:46 AM
3 Kudos
After upgrading HPD to 2.5 all our Sqoop jobs that imports "as-avrodatafile" fails with FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
at org.apache.sqoop.mapreduce.AvroOutputFormat.getRecordWriter(AvroOutputFormat.java:97)
at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:647)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Sqoop depends on avro version 1.8.0 (which does have that method) and I guess that the problem could be due to the presence of avro versions 1.7.4 (hbase) and 1.7.5 (hive). Please, is there a way to make yarn use version 1.8.0 instead of 1.7.x in case that is the problem?
... View more
Labels:
- Labels:
-
Apache Sqoop