Support Questions
Find answers, ask questions, and share your expertise

Distcp Issue error code 143

Distcp Issue error code 143

hadoop distcp source destination

Data Volume: 21 gb , Number of files ~2000

I am getting error : Error code 143.

I tried all combination like:

export HADOOP_HEAPSIZE=16096

export HADOOP_OPTS="-Xmx8G"

export HADOOP_CLIENT_OPTS="-Xms6096m -Xmx6096m"

hadoop distcp -D mapred.child.java.opts=-Xmx4096M -D mapreduce.map.memory.mb=-Xmx5096M -D ipc.client.fallback-to-simple-auth-allowed=true -D mapreduce.job.maxtaskfailures.per.tracker=1 source destination

As per distcp, It seems to be problem of heap memory due to large number of files meta data on name node. I can't change cluster setting by myself which has heap setup 1 gb. But I am passing heap property in the script, and it is not working.

But I am still getting error:

18/02/16 13:57:26 INFO tools.DistCp: DistCp job-id: job_1518700164032_0249

18/02/16 13:57:26 INFO mapreduce.Job: Running job: job_1518700164032_0249

18/02/16 13:57:32 INFO mapreduce.Job: Job job_1518700164032_0249 running in uber mode : false

18/02/16 13:57:32 INFO mapreduce.Job: map 0% reduce 0%

18/02/16 13:57:36 INFO mapreduce.Job: Task Id : attempt_1518700164032_0249_m_000018_0, Status : FAILED

Error: java.io.EOFException

at java.io.DataInputStream.readFully(DataInputStream.java:197)

at java.io.DataInputStream.readFully(DataInputStream.java:169)

at org.apache.hadoop.io.SequenceFile$Reader.sync(SequenceFile.java:2693)

at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:58)

at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)

18/02/16 13:57:36 INFO mapreduce.Job: Task Id : attempt_1518700164032_0249_m_000010_0, Status : FAILED

Error: java.io.EOFException

at java.io.DataInputStream.readFully(DataInputStream.java:197)

at java.io.DataInputStream.readFully(DataInputStream.java:169)

at org.apache.hadoop.io.SequenceFile$Reader.sync(SequenceFile.java:2693)

at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:58)

at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)