Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Will the MapReduce jobs fail when the NameNode crashes?

avatar

Lets assume a NameNode crashes in the HDFS HA (High Availability) setup and the Secondary/Fallback NameNode takes over, What will happen to the currently running MapReduce Jobs? Will they also fail?

> Also will the secondary NameNode takes over for the failed Primary NameNode automatically or do we need any administrator to manually redirect the client requests to the Secondary NameNode?

Thanks.

1 ACCEPTED SOLUTION

avatar
Master Guru
@Vinay Reddy Naguru

As answers given above - Your job should continue to run without any issue.

Regarding Automated failover - By default HDP has below property configured in hdfs-site.xml for automated failover.

 <property>
   <name>dfs.ha.automatic-failover.enabled</name>
   <value>true</value>
 </property>

If you want, you can do manual failover using below command ( run using root user )

sudo -u hdfs hdfs haadmin -failover nn1 nn2

Note - Above command will do failover from nn1 to nn2

You can read more information about automatic failover by referring to below link.

https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.ht...

View solution in original post

4 REPLIES 4

avatar
Guru

HA failover is automatic by default if you enabled failover from ambari. Mapreduce jobs won't fail during a failover scenario.

avatar

Job should continue to run unaffected by the NameNode failure.

Also, Standby NN should become active NN in HA mode.

avatar
Master Guru
@Vinay Reddy Naguru

As answers given above - Your job should continue to run without any issue.

Regarding Automated failover - By default HDP has below property configured in hdfs-site.xml for automated failover.

 <property>
   <name>dfs.ha.automatic-failover.enabled</name>
   <value>true</value>
 </property>

If you want, you can do manual failover using below command ( run using root user )

sudo -u hdfs hdfs haadmin -failover nn1 nn2

Note - Above command will do failover from nn1 to nn2

You can read more information about automatic failover by referring to below link.

https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.ht...

avatar
Rising Star

One point of clarification, the Secondary Name Node is not used for High Availability. It was poorly named and only provides checkpointing capabilities. You need to enable Name Node HA (which replaces the Secondary Name Node with a Standby Name Node) for failover to work. Ambari has a wizard to assist in enabling NameNode HA.

Once NameNode HA is enabled, jobs will continue if the Primary NameNode fails.