Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Distcp Issue error code 143

Distcp Issue error code 143

hadoop distcp source destination

Data Volume: 21 gb , Number of files ~2000

I am getting error : Error code 143.

I tried all combination like:

export HADOOP_HEAPSIZE=16096

export HADOOP_OPTS="-Xmx8G"

export HADOOP_CLIENT_OPTS="-Xms6096m -Xmx6096m"

hadoop distcp -D mapred.child.java.opts=-Xmx4096M -D mapreduce.map.memory.mb=-Xmx5096M -D ipc.client.fallback-to-simple-auth-allowed=true -D mapreduce.job.maxtaskfailures.per.tracker=1 source destination

As per distcp, It seems to be problem of heap memory due to large number of files meta data on name node. I can't change cluster setting by myself which has heap setup 1 gb. But I am passing heap property in the script, and it is not working.

But I am still getting error:

18/02/16 13:57:26 INFO tools.DistCp: DistCp job-id: job_1518700164032_0249

18/02/16 13:57:26 INFO mapreduce.Job: Running job: job_1518700164032_0249

18/02/16 13:57:32 INFO mapreduce.Job: Job job_1518700164032_0249 running in uber mode : false

18/02/16 13:57:32 INFO mapreduce.Job: map 0% reduce 0%

18/02/16 13:57:36 INFO mapreduce.Job: Task Id : attempt_1518700164032_0249_m_000018_0, Status : FAILED

Error: java.io.EOFException

at java.io.DataInputStream.readFully(DataInputStream.java:197)

at java.io.DataInputStream.readFully(DataInputStream.java:169)

at org.apache.hadoop.io.SequenceFile$Reader.sync(SequenceFile.java:2693)

at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:58)

at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)

18/02/16 13:57:36 INFO mapreduce.Job: Task Id : attempt_1518700164032_0249_m_000010_0, Status : FAILED

Error: java.io.EOFException

at java.io.DataInputStream.readFully(DataInputStream.java:197)

at java.io.DataInputStream.readFully(DataInputStream.java:169)

at org.apache.hadoop.io.SequenceFile$Reader.sync(SequenceFile.java:2693)

at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:58)

at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)

at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)

at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:422)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1865)

at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)

Don't have an account?
Coming from Hortonworks? Activate your account here