metron installation on existed ambari managed cluster

[#1]

i found the documentation of metron installation here :

https://cwiki.apache.org/confluence/display/METRON/Metron+Installation+on+an+Ambari-Managed+Cluster

but i have some question that not answered in documentation :

1. why it needs 2 master and what every nodes function?

2. what NN, HS, SN, RM, ZS, DN mean?

Thank You.

[#2 · Accepted solution]

Metron uses a number of Hadoop ecosystem components, and so tends to require separate master nodes for these for performance, this can also be used for resilience, though this diagram does not show full master HA.

To expand the abbreviations:-

NN = Name Node (the Hadoop HDFS name node stores file system meta data)

SN = Secondary Name Node (not very well named, but provides compaction and optimisation services for the NN)

RM = Resource Manager (the container coordinator which manages YARN resources and allocates them to running jobs)

ZS = Zookeeper Server (zookeeper is used extensively in Metron for storage and coordination of configuration. It is also used for similar purposes by many other Hadoop components)

DN = Data Node (this is an HDFS Data Node and responsible for storing the actual blocks in HDFS)

[#3]

@Simon Elliston Ball thank you for your answer, but what the difference in function of two master node ?

and what HS and NM mean. thank you

[#4]

@Bramantya Anggriawan, from the diagram you can see that the 2 master nodes are running different services. For instance Node 1 is running NN, SN, HS and RM services, while Node 2 is running ZS, HB Master (HBase Master), Storm UI, Nimbus etc. There is a need for two nodes in order to have optimum load balancing of the different services.

HS = History Server - which provides historical information about map-reduce jobs.

NM = Node Manager (http://hortonworks.com/blog/apache-hadoop-yarn-nodemanager)

[#5]

@asubramanian

@Simon Elliston Ball

Thanks, i have another question. in the requirement for hadoop and metron node it need 9 node (6 for hadoop and 3 for metron). but in the picture there is 11 node, and what is the specification for the other two node(ambari and metron installer node)?

Thanks.

[#6]

@asubramanian : Does the HS is Hive Server ? If not. Where is the optimum node to put Hive server ? I have pretty much lower resources in every node for testing purpose ( 8GB RAM & Dual core ).