Hi @sunile_manjee . Thank you very much for this excellent writeup and testing. I got the chance to run TeraGen, Terasort and TeraValidate with an environment which have around 40+ worker node with 3 master recently. Comparing to your results, it seems that for the Teragen mine is quite bad. My results: Teragen 57mnt 44sec, TeraSort 49mnt 01sec TeraValidate 4mnt16sec My spec: Masternode,cpu:16core , RAM:384GB Storage:12x2TB SATA 6Gb 7.2K RPM Workernode,cpu:24core, RAM;384GB Storage: 12x2TB NL SAS 12Gb 7.2k RPM network badwidth:20gbps My observation It believe the speed of the disk is very important here. Since my disk is only using SATA&SAS instead of SSD, I'm assuming this is the core reason why my Teragen results is bad even I got more nodes. Please correct me if this is not the case. But one thing that I saw from your BigStep test, the workernode is using local HDD not SSD like the AWS nodes. Shouldn't this decrease the TeraGen results? but it was the fastest one at 11min 49seconds. I also read from your test here https://community.cloudera.com/t5/Community-Articles/More-Hadoop-nodes-faster-IO-and-processing-time/ta-p/247155. The performance is way better using 5nodes against 3nodes. I wass hoping my 40+ node will top this, but it is not. Any advice is very very appreciated. P/s: I play with YARN configuration to try to get a better result. I set Map&Reduce cores&memory to 8 & 64GB. Do you think this help? Default setting (1core and 1gb ram)will make my terasort crash.
... View more
