Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Will the MapReduce jobs fail when the NameNode crashes?

Solved Go to solution

Will the MapReduce jobs fail when the NameNode crashes?

New Contributor

Lets assume a NameNode crashes in the HDFS HA (High Availability) setup and the Secondary/Fallback NameNode takes over, What will happen to the currently running MapReduce Jobs? Will they also fail?

> Also will the secondary NameNode takes over for the failed Primary NameNode automatically or do we need any administrator to manually redirect the client requests to the Secondary NameNode?

Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Will the MapReduce jobs fail when the NameNode crashes?

Super Guru
@Vinay Reddy Naguru

As answers given above - Your job should continue to run without any issue.

Regarding Automated failover - By default HDP has below property configured in hdfs-site.xml for automated failover.

 <property>
   <name>dfs.ha.automatic-failover.enabled</name>
   <value>true</value>
 </property>

If you want, you can do manual failover using below command ( run using root user )

sudo -u hdfs hdfs haadmin -failover nn1 nn2

Note - Above command will do failover from nn1 to nn2

You can read more information about automatic failover by referring to below link.

https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.ht...

4 REPLIES 4

Re: Will the MapReduce jobs fail when the NameNode crashes?

Guru

HA failover is automatic by default if you enabled failover from ambari. Mapreduce jobs won't fail during a failover scenario.

Re: Will the MapReduce jobs fail when the NameNode crashes?

Job should continue to run unaffected by the NameNode failure.

Also, Standby NN should become active NN in HA mode.

Re: Will the MapReduce jobs fail when the NameNode crashes?

Super Guru
@Vinay Reddy Naguru

As answers given above - Your job should continue to run without any issue.

Regarding Automated failover - By default HDP has below property configured in hdfs-site.xml for automated failover.

 <property>
   <name>dfs.ha.automatic-failover.enabled</name>
   <value>true</value>
 </property>

If you want, you can do manual failover using below command ( run using root user )

sudo -u hdfs hdfs haadmin -failover nn1 nn2

Note - Above command will do failover from nn1 to nn2

You can read more information about automatic failover by referring to below link.

https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.ht...

Re: Will the MapReduce jobs fail when the NameNode crashes?

Contributor

One point of clarification, the Secondary Name Node is not used for High Availability. It was poorly named and only provides checkpointing capabilities. You need to enable Name Node HA (which replaces the Secondary Name Node with a Standby Name Node) for failover to work. Ambari has a wizard to assist in enabling NameNode HA.

Once NameNode HA is enabled, jobs will continue if the Primary NameNode fails.