Support Questions
Find answers, ask questions, and share your expertise

with HDFS and Yarn with version 2.7.1.2.5 , do we still need MapReduce2

Explorer

In our Stack we installed with HDFS and Yarn with version 2.7.1.2.5.

Do we still need to install MapReduce2 ( which as two components like History Server and MapReduce2 Clients) . If so, On what nodes do we need to install MapReduce2 Clients ( like only on Data nodes or yarn hosts or MapReduce2 host only)

Regards

JJ

1 ACCEPTED SOLUTION

Accepted Solutions

Re: with HDFS and Yarn with version 2.7.1.2.5 , do we still need MapReduce2

Mentor

@Jacqualin jasmin

The simple answer is YES for MR

This whole depends on the number of nodes in your cluster and the desired setup HA or NOT. Usually, there are Masternodes ,Datanodes and edge node (Clients Only)

Masternode (if HA the recommended setup at least 3 nodes)

- NAMENODE. 
- JOURNALNODE. 
- ZKFC. 
- NODEMANAGER. 
- HBASE_MASTER(Optional). 
- HISTORYSERVER. 
- APP_TIMELINE_SERVER. 
- RESOURCEMANAGER. 
- ZOOKEEPER_SERVER.(at least 2 ) 
- JOURNALNODE. 
- MYSQL_SERVER. 
- ZKFC. 
- HIVE_SERVER. 
- HIVE_METASTORE. 
- WEBHCAT_SERVER. 
- ZOOKEEPER_CLIENT. 

Datanode (As desired)

- JOURNALNODE. 
- NODEMANAGER. 
- HBASE_REGIONSERVER(if Hbase installed). 
- ZOOKEEPER_CLIENT. 
- DATANODE 

Edgenode ( At least one)

- YARN_CLIENT. 
- ZOOKEEPER_CLIENT. 
- RANGER_ADMIN. 
- RANGER_USERSYNC. 
- METRICS_MONITOR. 
- METRICS_COLLECTOR. 
- ZEPPELIN_MASTER. 
- INFRA_SOLR. 
- INFRA_SOLR_CLIENT. 
- HBASE_CLIENT. 
- HDFS_CLIENT. 
- TEZ_CLIENT. 
- YARN_CLIENT. 
- HIVE_CLIENT. 
- MAPREDUCE2_CLIENT

In the above setup, some components are optional. See official Hortonworks document

View solution in original post

4 REPLIES 4

Re: with HDFS and Yarn with version 2.7.1.2.5 , do we still need MapReduce2

Guru

Hi @Jacqualin jasmin

Yes you should still install MapReduce for the history server, and the client tools. Certain components still use MapReduce under the hood, and having it installed does not usually cause harm or take up too many resources. The history server can be installed on any master node you like and makes sense to do so, while the clients are typically installed on edge (or data) nodes where users will have access.

Re: with HDFS and Yarn with version 2.7.1.2.5 , do we still need MapReduce2

Explorer

Thank Sonu, it helps me alot . keep up this spirit

Re: with HDFS and Yarn with version 2.7.1.2.5 , do we still need MapReduce2

Mentor

@Jacqualin jasmin

The simple answer is YES for MR

This whole depends on the number of nodes in your cluster and the desired setup HA or NOT. Usually, there are Masternodes ,Datanodes and edge node (Clients Only)

Masternode (if HA the recommended setup at least 3 nodes)

- NAMENODE. 
- JOURNALNODE. 
- ZKFC. 
- NODEMANAGER. 
- HBASE_MASTER(Optional). 
- HISTORYSERVER. 
- APP_TIMELINE_SERVER. 
- RESOURCEMANAGER. 
- ZOOKEEPER_SERVER.(at least 2 ) 
- JOURNALNODE. 
- MYSQL_SERVER. 
- ZKFC. 
- HIVE_SERVER. 
- HIVE_METASTORE. 
- WEBHCAT_SERVER. 
- ZOOKEEPER_CLIENT. 

Datanode (As desired)

- JOURNALNODE. 
- NODEMANAGER. 
- HBASE_REGIONSERVER(if Hbase installed). 
- ZOOKEEPER_CLIENT. 
- DATANODE 

Edgenode ( At least one)

- YARN_CLIENT. 
- ZOOKEEPER_CLIENT. 
- RANGER_ADMIN. 
- RANGER_USERSYNC. 
- METRICS_MONITOR. 
- METRICS_COLLECTOR. 
- ZEPPELIN_MASTER. 
- INFRA_SOLR. 
- INFRA_SOLR_CLIENT. 
- HBASE_CLIENT. 
- HDFS_CLIENT. 
- TEZ_CLIENT. 
- YARN_CLIENT. 
- HIVE_CLIENT. 
- MAPREDUCE2_CLIENT

In the above setup, some components are optional. See official Hortonworks document

View solution in original post

Re: with HDFS and Yarn with version 2.7.1.2.5 , do we still need MapReduce2

Explorer

Thank you very much Geoffrey, for your insights and supporting the community.