question Re: with HDFS and Yarn with version 2.7.1.2.5 , do we still need MapReduce2 in Support Questions

with HDFS and Yarn with version 2.7.1.2.5 , do we still need MapReduce2

Jacqueline — Thu, 21 Sep 2017 02:38:07 GMT

In our Stack we installed with HDFS and Yarn with version 2.7.1.2.5.

Do we still need to install MapReduce2 ( which as two components like History Server and MapReduce2 Clients) . If so, On what nodes do we need to install MapReduce2 Clients ( like only on Data nodes or yarn hosts or MapReduce2 host only)

Regards

Re: with HDFS and Yarn with version 2.7.1.2.5 , do we still need MapReduce2

ssahi — Thu, 21 Sep 2017 03:34:44 GMT

Hi @Jacqualin jasmin

Yes you should still install MapReduce for the history server, and the client tools. Certain components still use MapReduce under the hood, and having it installed does not usually cause harm or take up too many resources. The history server can be installed on any master node you like and makes sense to do so, while the clients are typically installed on edge (or data) nodes where users will have access.

Re: with HDFS and Yarn with version 2.7.1.2.5 , do we still need MapReduce2

Shelton — Thu, 21 Sep 2017 03:35:31 GMT

@Jacqualin jasmin

The simple answer is YES for MR

This whole depends on the number of nodes in your cluster and the desired setup HA or NOT. Usually, there are Masternodes ,Datanodes and edge node (Clients Only)

Masternode (if HA the recommended setup at least 3 nodes)

- NAMENODE. 
- JOURNALNODE. 
- ZKFC. 
- NODEMANAGER. 
- HBASE_MASTER(Optional). 
- HISTORYSERVER. 
- APP_TIMELINE_SERVER. 
- RESOURCEMANAGER. 
- ZOOKEEPER_SERVER.(at least 2 ) 
- JOURNALNODE. 
- MYSQL_SERVER. 
- ZKFC. 
- HIVE_SERVER. 
- HIVE_METASTORE. 
- WEBHCAT_SERVER. 
- ZOOKEEPER_CLIENT.

Datanode (As desired)

- JOURNALNODE. 
- NODEMANAGER. 
- HBASE_REGIONSERVER(if Hbase installed). 
- ZOOKEEPER_CLIENT. 
- DATANODE

Edgenode ( At least one)

- YARN_CLIENT. 
- ZOOKEEPER_CLIENT. 
- RANGER_ADMIN. 
- RANGER_USERSYNC. 
- METRICS_MONITOR. 
- METRICS_COLLECTOR. 
- ZEPPELIN_MASTER. 
- INFRA_SOLR. 
- INFRA_SOLR_CLIENT. 
- HBASE_CLIENT. 
- HDFS_CLIENT. 
- TEZ_CLIENT. 
- YARN_CLIENT. 
- HIVE_CLIENT. 
- MAPREDUCE2_CLIENT

In the above setup, some components are optional. See official Hortonworks document

Re: with HDFS and Yarn with version 2.7.1.2.5 , do we still need MapReduce2

Jacqueline — Fri, 22 Sep 2017 00:04:27 GMT

Thank you very much Geoffrey, for your insights and supporting the community.

Re: with HDFS and Yarn with version 2.7.1.2.5 , do we still need MapReduce2

Jacqueline — Fri, 22 Sep 2017 00:05:26 GMT

Thank Sonu, it helps me alot . keep up this spirit