Teragen, Terasort, and Teravalidate Performance testing on AWS

Short Description:

Teragen and Terasort Performance testing on AWS

Article

This article should be used with extreme care. Do not use as benchmark. I performed this test to simply run a quick 1 Terabype teragen test on AWS to determine what type of performance I can get from mapreduce on AWS with VERY LITTLE configuration tweaking/tuning

On my github page here you will find the following:

• teragen script
• hadoop,yarn,mapred,capacity scheduler configurations used during testing

Hardware: (Master & Datanode)

1 Master, 3 Data nodes

d2.4xlarge, 16vCPU, 122GB ram, (max) 12x2000 Storage

TeraGen Results:

1hrs, 6mins, 38sec

Job Counters:

Terasort Results:

1hrs, 34mins, 20sec

Teravalidate Results:

25mins, 27sec