Is Secondary Name node is mandatory for any distributed hadoop cluster?

[#1]

A HA namenode is not mandatory but highly encouraged. You cannot perform rolling upgrades without a HA namenode configured.

[#2]

Secondary NameNode is mandatory in every Hadoop installation. HA NameNode is higly suggested, but is something different from Seconday NameNode. Since Hadoop v2 release you can easily deploy a HA config of the NameNode with an Active and Standby NameNode (requires ZooKeeper). If you've a really small lab cluster you can avoid the HA NameNode config.

[#3]

for really small test environments, you can disable secondary namenode, our sandbox does not have secondary namenode running.

[#4 · Accepted solution]

If you are running an HA NameNode using Quorum Journal Manager, then running the SecondaryNameNode is not required. Actually, it would be incorrect to deploy a SecondaryNameNode alongside an HA NameNode pair.

Before implementation of HA with Quorum Journal Manager, the function of the SecondaryNameNode was to create a periodic checkpoint (a new fsimage file) of the NameNode metadata and upload it back to the NameNode. Without checkpointing, the NameNode's edit log would grow continuously. A very large edit log is problematic, because it slows down NameNode restarts. Replaying a large edit log is much more time-consuming than loading a recent metadata checkpoint and applying a small edit log on top of it.

With an HA deployment, the standby NameNode in the pair takes over the responsibility of periodic checkpointing previously performed by the SecondaryNameNode. Therefore, it is unnecessary (and invalid) to run a SecondaryNameNode. If you choose not to deploy with HA for some reason, then the SecondaryNameNode is recommended so that you get periodic checkpoints.

There is more discussion of this in the Apache documentation on NameNode HA using Quorum Journal Manager, particularly the section on hardware selection.

http://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.htm...

[#5]

Thank you all for your answers.