Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

2 ambari clusters in one big cluster

avatar

hi all


I want to hear from hortonworks guys about the following idea


we want to build the following architecture


1. ambari cluster that include ( HDP 3.1 )

3-5 kafka machines

3 zookeper servers that are dedicated only for the kafka machines


2. ambari cluster that include all other application ( HDP 3.1 )

HDFS - service

YARN - service

MAPREDUCE - service

HIVE - service

SPARK

3 Zookeepers that dedicated to hdfs , yarn , etc

data nodes machines


so both cluster should talk with each other like one big cluster


is this possible ?


Michael-Bronson
1 ACCEPTED SOLUTION

avatar
Master Mentor

@Michael Bronson

Regarding your query: "2 ambari clusters in one big cluster"

Ambari can currently manage only one cluster deployed to it. So from ambari perspective you manage two cluster from same Ambari.

However, You can run two clusters running separately and still be able to establish communication between Kafka cluster to the other cluster which has all the services running.

Like Spark running in Cluster two can interact with External Kafka Broker running inside Cluster one.

View solution in original post

4 REPLIES 4

avatar
Master Mentor

@Michael Bronson

Regarding your query: "2 ambari clusters in one big cluster"

Ambari can currently manage only one cluster deployed to it. So from ambari perspective you manage two cluster from same Ambari.

However, You can run two clusters running separately and still be able to establish communication between Kafka cluster to the other cluster which has all the services running.

Like Spark running in Cluster two can interact with External Kafka Broker running inside Cluster one.

avatar

@DEAR Jay "clusters running separately and still be able to establish communication between Kafka cluster to the other cluster which has all the services "

can you please explain in basic details what are the configuration on both cluster to enable contamination between one ambari cluster to the ambari cluster that include the kafka?

Michael-Bronson

avatar
Master Mentor

@Michael Bronson

What i mean to say is this ::: At AmbariServer level you can not manage two clusters as a Single cluster.

But if you want to Produce Messages to Kafka (For example via Spark Application then that can be done even in Non Ambari Managed cluster as well right)

For Example: In your Spark application define the correct Kafka Broker host/port ...etc like following:

https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html



avatar

The above question and the entire reply thread below was originally posted in the Community Help track. On Tue Jul 2 01:01 UTC 2019, a member of the HCC moderation staff moved it to the Design & Architecture track. The Community Help Track is intended for questions about using the HCC site itself, not technical questions.

Bill Brooks, Community Moderator
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.