Hi, We have written a mapreduce job to process log files. As of now we have around 52GB of input files but it is taking around an hour to process the data.It creates only one reducer job by default.Often we get to see a timeout error in the reduce task and then it restarts and gets completed. Below is the stats for the successful completion of the job. Kindly let us know how the performance can be improved. File System Counters FILE: Number of bytes read=876100387 FILE: Number of bytes written=1767603407 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=52222279591 HDFS: Number of bytes written=707429882 HDFS: Number of read operations=351 HDFS: Number of large read operations=0 HDFS: Number of write operations=2 Job Counters Failed reduce tasks=1 Launched map tasks=116 Launched reduce tasks=2 Other local map tasks=116 Total time spent by all maps in occupied slots (ms)=9118125 Total time spent by all reduces in occupied slots (ms)=7083783 Total time spent by all map tasks (ms)=3039375 Total time spent by all reduce tasks (ms)=2361261 Total vcore-seconds taken by all map tasks=3039375 Total vcore-seconds taken by all reduce tasks=2361261 Total megabyte-seconds taken by all map tasks=25676640000 Total megabyte-seconds taken by all reduce tasks=20552415744 Map-Reduce Framework Map input records=49452982 Map output records=5730971 Map output bytes=864140911 Map output materialized bytes=876101077 Input split bytes=13922 Combine input records=0 Combine output records=0 Reduce input groups=1082133 Reduce shuffle bytes=876101077 Reduce input records=5730971 Reduce output records=5730971 Spilled Records=11461942 Shuffled Maps =116 Failed Shuffles=0 Merged Map outputs=116 GC time elapsed (ms)=190633 CPU time spent (ms)=4536110 Physical memory (bytes) snapshot=340458307584 Virtual memory (bytes) snapshot=1082745069568 Total committed heap usage (bytes)=378565820416 Shuffle Errors BAD_ID=0 CONNECTION=0 IO_ERROR=0 WRONG_LENGTH=0 WRONG_MAP=0 WRONG_REDUCE=0 File Input Format Counters Bytes Read=52222265669 File Output Format Counters Bytes Written=707429882 ... View more