Support Questions

Find answers, ask questions, and share your expertise

Map reduce jobs getting fail

while loading 30 gb file to hbase using map reduce program, 29 mappers are launched and failed.

errors :

Error: java.lang.ArrayIndexOutOfBoundsException at java.util.zip.CRC32.update(CRC32.java:74) at org.apache.hadoop.util.DataChecksum.update(DataChecksum.java:265) at org.apache.hadoop.mapred.IFileOutputStream.write(IFileOutputStream.java:87) at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:58) at java.io.DataOutputStream.write(DataOutputStream.java:107) at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:245) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1621) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1489) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143

what could be reason of getting fail mapred program ?

4 REPLIES 4

@Anurag Mishra

The issue could be due to incorrect value for io.sort.mb. Try reducing the value for the property and run the program again.

@Sindhu could you please explain the reason, and little more about io.sort.mb , and why is it recommended to reduce the value ?

as i see io.sort.mb : The total amount of buffer memory to use while sorting files, in megabytes, so if i reduce value then efficiency will go down? i could not understand reducing the value for this property

yarn.scheduler.minimum-allocation-mb = 1024

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.