Hi, went through my testing again.  Unfortunately, I missed the step where I need to change/add the parameters on my commandline. Change my TeraGen, TeraSort and TeraValidate parameter and got better results   TeraGen: 1 min 57 sec TeraSort: 22min 55sec TeraValidate: 1 min 23sec.   Thank you very much for your writeup again.  ... View more

Hi @sunile_manjee . Thank you very much for this excellent writeup and testing.   I got the chance to run TeraGen, Terasort and TeraValidate with an environment which have around 40+ worker node with 3 master recently. Comparing to your results, it seems that for the Teragen mine is quite bad.    My results: Teragen 57mnt 44sec, TeraSort 49mnt 01sec TeraValidate 4mnt16sec   My spec: Masternode,cpu:16core , RAM:384GB Storage:12x2TB SATA 6Gb 7.2K RPM Workernode,cpu:24core, RAM;384GB Storage: 12x2TB NL SAS  12Gb 7.2k RPM network badwidth:20gbps   My observation It believe the speed of the disk is very important here. Since my disk is only using SATA&SAS instead of SSD, I'm assuming this is the core reason why my Teragen results is bad even I got more nodes. Please correct me if this is not the case. But one thing that I saw from your BigStep test, the workernode is using local HDD not SSD like the AWS nodes. Shouldn't this decrease the TeraGen results? but it was the fastest one at 11min 49seconds.   I also read from your test here https://community.cloudera.com/t5/Community-Articles/More-Hadoop-nodes-faster-IO-and-processing-time/ta-p/247155. The performance is way better using 5nodes against 3nodes. I wass hoping my 40+ node will top this, but it is not. Any advice is very very appreciated.   P/s: I play with YARN configuration to try to get a better result. I set Map&Reduce cores&memory to 8 & 64GB. Do you think this help? Default setting (1core and 1gb ram)will make my terasort crash. ... View more

Hi @StevenOD  Thank you very much for your info. I've actually configured CDP-DC at my office to try it out. It is very small scale setup for now and currently only have one base cluster.   I'm trying to create a project using similar to use case shown in very recent Cloudera Data Flow webinar which is injecting/streaming iot sensors data to data warehouse using NiFi and Kafka.   The problem is, in my CDP-DC environment, there is no option to create a cluster from templates like the one available in CDP Public Cloud such as Streaming Messaging and Flow Management template which natively consist component like NiFi. For now there is only Data Engineering, Data Mart,Operational Database & custom cluster template available.   So it is safe for me to say that all these cluster template and components available in public cloud will be available to CDP Private cloud soon right? Or I can proceed with my CDP-DC and install NiFi through csd to proceed with my project. Really appreciate if you could point me to the right direction. Thanks again ... View more

Is this Cloudera Management Console available to be install on premise? I would really like to try it. I did some reading, and what I understand, it is currently only on Public Cloud right? Is my understanding correct? Thanks ... View more