Will the MapReduce jobs fail when the NameNode crashes?

[#1]

Lets assume a NameNode crashes in the HDFS HA (High Availability) setup and the Secondary/Fallback NameNode takes over, What will happen to the currently running MapReduce Jobs? Will they also fail?

> Also will the secondary NameNode takes over for the failed Primary NameNode automatically or do we need any administrator to manually redirect the client requests to the Secondary NameNode?

Thanks.

[#2]

HA failover is automatic by default if you enabled failover from ambari. Mapreduce jobs won't fail during a failover scenario.

[#3]

Job should continue to run unaffected by the NameNode failure.

Also, Standby NN should become active NN in HA mode.

[#4 · Accepted solution]

@Vinay Reddy Naguru

As answers given above - Your job should continue to run without any issue.

Regarding Automated failover - By default HDP has below property configured in hdfs-site.xml for automated failover.

 <property>
   <name>dfs.ha.automatic-failover.enabled</name>
   <value>true</value>
 </property>

If you want, you can do manual failover using below command ( run using root user )

sudo -u hdfs hdfs haadmin -failover nn1 nn2

Note - Above command will do failover from nn1 to nn2

You can read more information about automatic failover by referring to below link.

https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.ht...

[#5]

One point of clarification, the Secondary Name Node is not used for High Availability. It was poorly named and only provides checkpointing capabilities. You need to enable Name Node HA (which replaces the Secondary Name Node with a Standby Name Node) for failover to work. Ambari has a wizard to assist in enabling NameNode HA.

Once NameNode HA is enabled, jobs will continue if the Primary NameNode fails.