question Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production? in Archives of Support Questions (Read Only)

Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

amcbarnett — Thu, 21 Jan 2016 00:56:59 GMT

What are best practices for Deploying Storm Components on a cluster for scalablity and growth? We are thinking of having dedicated nodes for Storm on YARN. Also would anything go on an edge node?

For example in a cluster, the thought is to have three Storm nodes (S1, S2, S3) dedicated with the following allocations:

Storm Nimbus:

Choose S1 as Storm Master to deploy Storm Nimbus... Or Probably a best practice is to not co-locate Nimbus with any worker node
S1 node will also have the Storm UI

Storm Supervisors/ Workers

Choose S1, S2, S3 to deploy Storm Supervisors

Zookeeper Cluster

Since Kafka is usually used with Storm, have a separate Zookeeper cluster for Kafka and Storm.
DON"T put the Zookeeper cluster on the Kafka nodes (K1, K2, K3).
Put the Zookeeper on the Storm nodes (S1, S2, S3)

Storm UI

Will be on the same node as the Nimbus: S1 or Edge

DRPC Server

What is the best practice to place this?

So in Summary, if we have three dedicated nodes for Storm, the thinking is to allocate as follows:

S1 Node:

Storm Nimbus/ Storm UI (Maybe it is not a best practice to put Storm Nimbus on worker nodes and put this on Edge node?)
Storm Supervisor
Zookeeper

S2 Node:

Storm Supervisor
Zookeeper

S3 Node:

Storm Supervisor
Zookeeper

Edge Node:

Storm Nimbus/ Storm UI (Maybe it is not a best practice to put Storm Nimbus on worker nodes and put this on Edge node?)

Finally would the DRPC go on the Nimbus node? Any thoughts on this? Am I on the right track? Would anything go on an edge node?

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

aervits — Thu, 21 Jan 2016 01:02:18 GMT

storm supervisors on their own nodes, kafka brokers can be collocated with datanodes, that's my findings from our recent POC. I can give you more detail on phone. @Ancil McBarnett

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

aervits — Thu, 21 Jan 2016 01:02:19 GMT

@Ancil McBarnett Have you looked at this guide?

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

amcbarnett — Thu, 21 Jan 2016 01:02:20 GMT

Yes, but it does not go into deployment from a cluster topology point of view (except for the discussion on zookeeper) @Wes Floyd

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

pminovic — Thu, 21 Jan 2016 14:14:52 GMT

Hi @Ancil McBarnett my 2 cents:

Nothing on Edge nodes, you have no idea what the guys will do there
Nimbus, Storm UI and DRPC on one of cluster master nodes. If this is a stand-alone Storm&Kafka cluster then set a master and put these guys together with Ambari there.
Supervisors on dedicated nodes. In hdfs cluster you can collocate them with Data nodes.
Dedicated Kafka broker nodes, but see below
Dedicated ZK for Kafka, however in case of Kafka-0.8.2 or higher, if consumers don't keep offsets on ZK, low to medium traffic and you have at least 3 brokers,then you can start with collocating Kafka ZK with brokers. In this case ZK should use dedicated disk.

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

amcbarnett — Fri, 22 Jan 2016 03:31:54 GMT

@Predrag Minovic why not Supervisors on their own nodes, not on Data nodes?

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

aervits — Fri, 22 Jan 2016 03:32:48 GMT

@Ancil McBarnett @tgoetz suggests to put supervisors on their own.

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

phargis — Fri, 22 Jan 2016 06:52:58 GMT

This is not a complete answer, but would like to also add that, by default, Kafka brokers write to local storage (not HDFS), and therefore, benefit from fast local disk (SSD) and/or multiple spindles to parallelize writes to partitions. I don't know of a formula to calculate this, but try to maximize I/O throughput to disk, and allocate # spindles up to the # of available CPUs per node. Lots of Hadoop architectures don't really specify allocation for local storage (beyond OS disk), and therefore it is something to be aware of.

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

pminovic — Fri, 22 Jan 2016 07:08:43 GMT

@Ancil McBarnett Oh yes, supervisors definitely on dedicated nodes if you have enouhg nodes. I updated my answer.

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

aervits — Wed, 03 Feb 2016 00:24:59 GMT

@Ancil McBarnett accept best answer 🙂

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

amcbarnett — Fri, 05 Feb 2016 00:43:24 GMT

As per

https://community.hortonworks.com/content/kbentry/550/unofficial-storm-and-kafka-best-practices-guide.html

ZK on separate nodes from Kafka Broker. Do Not Install zk nodes on the same node as kafka broker if you want optimal Kafka performance. Disk I/O both kafka and zk are disk I/O intensive

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

amcbarnett — Mon, 19 Aug 2019 11:54:57 GMT

This is the picture I have come up with

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

pminovic — Fri, 05 Feb 2016 11:26:22 GMT

Cute pic! And everything looks good, however, even in the "Light Storm, Kafka" case I'd create a dedicated Kafka ZooKeeper and put it on Kafka brokers, so that ZKs write to their own disks (for example, 5 disks for Kafka, 1 for ZK).

Re: Best Practices for Storm Deployment on a Hadoop Cluster using Ambari. How would you allocate components in production?

baannee — Fri, 31 Mar 2017 19:21:11 GMT

Hi, I am planning to create Ambari Hadoop Storm Cluster and as this is fresh new for me I have some doubts how to setup it on the best way. Here is what I have for resources:

- Platform: AWS (8 EC2 instances - 1 master. 4 slaves, 3 workers (zookeepers))

- Tool: As I want to automate setup, I will use Terraform, Ansible and Blueprint to setup all environment

- I research a little bit and create some conclusion and I need some advice/opinion is this a good path???

Thanks

MASTER	SLAVE	ZOO
NAMENODE	SECONDARY_NAMENODE	DATANODE
NIMBUS	RESOURCE_MANAGER	NODEMANAGER
DRPC_SERVER	SUPERVISOR	ZOOKEEPER_SERVER
STORM_UI_SERVER	ZOOKEEPER_CLIENT	METRICS_MONITOR
ZOOKEEPER_CLIENT	METRICS_MONITOR	MAPREDUCE2_CLIENT
HDFS_CLIENT	HDFS_CLIENT	HDFS_CLIENT
PIG	PIG	PIG
TEZ_CLIENT	TEZ_CLIENT	TEZ_CLIENT
YARN_CLIENT	YARN_CLIENT	YARN_CLIENT
METRICS_COLLECTOR	HISTORY_SERVER
METRICS_GRAFANA	MAPREDUCE2_CLIENT
APP_TIMELINE_SERVER
HIVE_SERVER	HCAT
HIVE_METASTORE	WEBHCAT_SERVER
MYSQL_SERVER	HIVE_CLIENT