2 ambari clusters in one big cluster

[#1]

hi all

I want to hear from hortonworks guys about the following idea

we want to build the following architecture

1. ambari cluster that include ( HDP 3.1 )

3-5 kafka machines

3 zookeper servers that are dedicated only for the kafka machines

2. ambari cluster that include all other application ( HDP 3.1 )

HDFS - service

YARN - service

MAPREDUCE - service

HIVE - service

SPARK

3 Zookeepers that dedicated to hdfs , yarn , etc

data nodes machines

so both cluster should talk with each other like one big cluster

is this possible ?

[#2 · Accepted solution]

@Michael Bronson

Regarding your query: "2 ambari clusters in one big cluster"

Ambari can currently manage only one cluster deployed to it. So from ambari perspective you manage two cluster from same Ambari.

However, You can run two clusters running separately and still be able to establish communication between Kafka cluster to the other cluster which has all the services running.

Like Spark running in Cluster two can interact with External Kafka Broker running inside Cluster one.

[#3]

@DEAR Jay "clusters running separately and still be able to establish communication between Kafka cluster to the other cluster which has all the services "

can you please explain in basic details what are the configuration on both cluster to enable contamination between one ambari cluster to the ambari cluster that include the kafka?

[#4]

@Michael Bronson

What i mean to say is this ::: At AmbariServer level you can not manage two clusters as a Single cluster.

But if you want to Produce Messages to Kafka (For example via Spark Application then that can be done even in Non Ambari Managed cluster as well right)

For Example: In your Spark application define the correct Kafka Broker host/port ...etc like following:

https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html

[#5]

The above question and the entire reply thread below was originally posted in the Community Help track. On Tue Jul 2 01:01 UTC 2019, a member of the HCC moderation staff moved it to the Design & Architecture track. The Community Help Track is intended for questions about using the HCC site itself, not technical questions.