Sqoop import to avro failing - which jars to be used ?

kaliyugantagoni — Tue, 11 Oct 2016 16:36:10 GMT

HDP-2.5.0.0 using Ambari 2.4.0.1

A Sqoop import to avro fails with the following error :

16/10/11 08:26:32 INFO mapreduce.Job: Job job_1476162030393_0002 running in uber mode : false
16/10/11 08:26:32 INFO mapreduce.Job:  map 0% reduce 0%
16/10/11 08:26:40 INFO mapreduce.Job:  map 25% reduce 0%
16/10/11 08:26:40 INFO mapreduce.Job: Task Id : attempt_1476162030393_0002_m_000001_0, Status : FAILED
Error: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
16/10/11 08:26:40 INFO mapreduce.Job: Task Id : attempt_1476162030393_0002_m_000000_0, Status : FAILED
Error: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
16/10/11 08:26:40 INFO mapreduce.Job: Task Id : attempt_1476162030393_0002_m_000003_0, Status : FAILED
Error: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
16/10/11 08:26:41 INFO mapreduce.Job:  map 0% reduce 0%
16/10/11 08:26:42 INFO mapreduce.Job: Task Id : attempt_1476162030393_0002_m_000002_0, Status : FAILED
Error: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
16/10/11 08:26:46 INFO mapreduce.Job: Task Id : attempt_1476162030393_0002_m_000001_1, Status : FAILED
Error: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
16/10/11 08:26:47 INFO mapreduce.Job: Task Id : attempt_1476162030393_0002_m_000000_1, Status : FAILED
Error: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
16/10/11 08:26:47 INFO mapreduce.Job: Task Id : attempt_1476162030393_0002_m_000003_1, Status : FAILED
Error: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
16/10/11 08:26:48 INFO mapreduce.Job: Task Id : attempt_1476162030393_0002_m_000002_1, Status : FAILED
Error: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
16/10/11 08:26:51 INFO mapreduce.Job: Task Id : attempt_1476162030393_0002_m_000001_2, Status : FAILED
Error: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
16/10/11 08:26:51 INFO mapreduce.Job: Task Id : attempt_1476162030393_0002_m_000002_2, Status : FAILED
Error: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
16/10/11 08:26:51 INFO mapreduce.Job: Task Id : attempt_1476162030393_0002_m_000003_2, Status : FAILED
Error: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
16/10/11 08:26:52 INFO mapreduce.Job: Task Id : attempt_1476162030393_0002_m_000000_2, Status : FAILED
Error: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
16/10/11 08:26:57 INFO mapreduce.Job:  map 100% reduce 0%
16/10/11 08:26:57 INFO mapreduce.Job: Job job_1476162030393_0002 failed with state FAILED due to: Task failed task_1476162030393_0002_m_000002
Job failed as tasks failed. failedMaps:1 failedReduces:0

The YARN application log ends with :

FATAL [main] org.apache.hadoop.mapred.YarnChild: Error running child : java.lang.NoSuchMethodError: org.apache.avro.reflect.ReflectData.addLogicalTypeConversion(Lorg/apache/avro/Conversion;)V
        at org.apache.sqoop.mapreduce.AvroOutputFormat.getRecordWriter(AvroOutputFormat.java:97)
        at org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:647)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

The original installation had the following libraries under /usr/hdp/2.5.0.0-1245/sqoop/lib:

avro-mapred-1.8.0-hadoop2.jar, parquet-avro-1.4.1.jar, avro-1.8.0.jar

I tried first replacing(ONLY one jar at a time under the lib) avro-mapred-1.8.0-hadoop2.jar with avro-mapred-1.8.1-hadoop2.jar and avro-mapred-1.7.7-hadoop2.jar. When that didn't help, I tried using the jars from the HDP 2.4 distribution viz. avro-1.7.5.jar and avro-mapred-1.7.5-hadoop2.jar, yet the error persisted.

How shall I fix the error ?

Re: Sqoop import to avro failing - which jars to be used ?

ssubhas — Tue, 11 Oct 2016 16:51:20 GMT

@Kaliyug Antagonist

This is actually a known issue, and there is a Jira for a documentation bug to get this fixed in a later HDP release. Sqoop uses 1.8.0 of avro and there are other Hadoop components using 1.7.5 or 1.7.4 avro.

Please add the following property after 'import': -Dmapreduce.job.user.classpath.first=true

Example:

sqoop import -Dmapreduce.job.user.classpath.first=true -Dhadoop.security.credential.provider.path=jceks://x.jceks --connect jdbc:db2://xxx:60000/x2 --username xx -password-alias xx --as-avrodatafile --target-dir xx/data/test --fields-terminated-by '\001' --table xx -m 1

Re: Sqoop import to avro failing - which jars to be used ?

kaliyugantagoni — Tue, 11 Oct 2016 17:35:49 GMT

It worked :) Can you provide the JIRA bug link ?

Re: Sqoop import to avro failing - which jars to be used ?

ejunior76 — Mon, 06 Feb 2017 07:50:17 GMT

Thank you Sindhu,

I was facing the same problem. Now it works.

Re: Sqoop import to avro failing - which jars to be used ?

sarvesh_sood — Sun, 26 Nov 2017 09:56:00 GMT

Thanks! Even i faced similar issue.

Re: Sqoop import to avro failing - which jars to be used ?

jun251111 — Sat, 17 Mar 2018 21:46:58 GMT

Thank you.

Re: Sqoop import to avro failing - which jars to be used ?

muhammad_danish — Mon, 02 Apr 2018 03:08:51 GMT

Thanks It worked !

could you please explain this addition how is this making it work ? " -Dmapreduce.job.user.classpath.first=true -Dhadoop.security.credential.provider.path=jceks://x.jceksor "

Re: Sqoop import to avro failing - which jars to be used ?

manmeetkaur_ran — Thu, 09 May 2019 04:58:54 GMT

For HDP 3.1, setting the following property will resolve the issue:

-Dmapreduce.job.classloader=true

Re: Sqoop import to avro failing - which jars to be used ?

Praveenya — Tue, 19 Jan 2021 11:38:30 GMT

Thank you So much Subha, It worked like magic.

question Re: Sqoop import to avro failing - which jars to be used ? in Support Questions

Sqoop import to avro failing - which jars to be used ?

Re: Sqoop import to avro failing - which jars to be used ?

Re: Sqoop import to avro failing - which jars to be used ?

Re: Sqoop import to avro failing - which jars to be used ?

Re: Sqoop import to avro failing - which jars to be used ?

Re: Sqoop import to avro failing - which jars to be used ?

Re: Sqoop import to avro failing - which jars to be used ?

Re: Sqoop import to avro failing - which jars to be used ?

Re: Sqoop import to avro failing - which jars to be used ?