Sqoop import as-avrodatafile fails with java.lang.NoSuchMethodError: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion

[#1]

After upgrading HPD to 2.5 all our Sqoop jobs that imports "as-avrodatafile" fails with

FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
        at org.apache.sqoop.mapreduce.AvroOutputFormat.getRecordWriter(AvroOutputFormat.java:97)
        at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:647)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Sqoop depends on avro version 1.8.0 (which does have that method) and I guess that the problem could be due to the presence of avro versions 1.7.4 (hbase) and 1.7.5 (hive).

Please, is there a way to make yarn use version 1.8.0 instead of 1.7.x in case that is the problem?

[#2 · Accepted solution]

@Sergio Aparicio

This is actually a known issue, and there is a Jira for a documentation bug to get this fixed in a later HDP release. Sqoop uses 1.8.0 of avro and there are other Hadoop components using 1.7.5 or 1.7.4 avro. Please add the following property after 'import': -Dmapreduce.job.user.classpath.first=true Example: sqoop import -Dmapreduce.job.user.classpath.first=true -Dhadoop.security.credential.provider.path=jceks://x.jceks --connect jdbc:db2://xxx:60000/VKTXAP02 --username xx -password-alias xx --as-avrodatafile --target-dir xx/data/test --fields-terminated-by '\001' --table xx -m 1

[#3]

Thank you very much, with -Dmapreduce.job.user.classpath.first=true works like a charm!

[#4]

I'm getting this same error , but when trying to call swoop export. The -Dmapreduce.job.user.classpath.first=true is not valid for export. Does anyone know how to resolve this similar issue but for exports. (I'm running HDP 2.6)

[#5]

Sorry, never mind -- I didn't realize that the -D parameters must appear before other params. I added it as my first param and it worked like a charm.

[#6]

this solution is not working for please tell me where I am going wrong

sqoop-import -Dmapreduce.job.user.classpath.first=true -Dhadoop.security.credential.provider.path=jceks://x.jceks \
--connect="jdbc:mysql://quickstart.cloudera:3306/retail_db" \
--username retail_dba \
--password cloudera \
--table=departments \
--hive-import \
--target-dir=/departments \
--as-avrodatafile