Skip to main content
mike_bronson7
mike_bronson7
Gold ⭐️⭐️⭐️
July 2, 2019
Solved

2 ambari clusters in one big cluster

hi all


I want to hear from hortonworks guys about the following idea


we want to build the following architecture


1. ambari cluster that include ( HDP 3.1 )

3-5 kafka machines

3 zookeper servers that are dedicated only for the kafka machines


2. ambari cluster that include all other application ( HDP 3.1 )

HDFS - service

YARN - service

MAPREDUCE - service

HIVE - service

SPARK

3 Zookeepers that dedicated to hdfs , yarn , etc

data nodes machines


so both cluster should talk with each other like one big cluster


is this possible ?


jsensharma
jsensharmaAnswer
Platinum ⭐️⭐️
July 2, 2019

@Michael Bronson

Regarding your query: "2 ambari clusters in one big cluster"

Ambari can currently manage only one cluster deployed to it. So from ambari perspective you manage two cluster from same Ambari.

However, You can run two clusters running separately and still be able to establish communication between Kafka cluster to the other cluster which has all the services running.

Like Spark running in Cluster two can interact with External Kafka Broker running inside Cluster one.

View in thread

4 replies

jsensharma
Platinum ⭐️⭐️
jsensharmaAnswer
Platinum ⭐️⭐️
July 2, 2019

@Michael Bronson

Regarding your query: "2 ambari clusters in one big cluster"

Ambari can currently manage only one cluster deployed to it. So from ambari perspective you manage two cluster from same Ambari.

However, You can run two clusters running separately and still be able to establish communication between Kafka cluster to the other cluster which has all the services running.

Like Spark running in Cluster two can interact with External Kafka Broker running inside Cluster one.

Translate
    mike_bronson7
    mike_bronson7Author
    Gold ⭐️⭐️⭐️
    July 2, 2019

    @DEAR Jay "clusters running separately and still be able to establish communication between Kafka cluster to the other cluster which has all the services "

    can you please explain in basic details what are the configuration on both cluster to enable contamination between one ambari cluster to the ambari cluster that include the kafka?

    Translate
    Michael-Bronson
    jsensharma
    Platinum ⭐️⭐️
    jsensharma
    Platinum ⭐️⭐️
    July 2, 2019

    @Michael Bronson

    What i mean to say is this ::: At AmbariServer level you can not manage two clusters as a Single cluster.

    But if you want to Produce Messages to Kafka (For example via Spark Application then that can be done even in Non Ambari Managed cluster as well right)

    For Example: In your Spark application define the correct Kafka Broker host/port ...etc like following:

    https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html



    Translate
    ask_bill_brooks
    Platinum ⭐️
    ask_bill_brooks
    Platinum ⭐️
    July 2, 2019

    The above question and the entire reply thread below was originally posted in the Community Help track. On Tue Jul 2 01:01 UTC 2019, a member of the HCC moderation staff moved it to the Design & Architecture track. The Community Help Track is intended for questions about using the HCC site itself, not technical questions.

    Translate
    Bill Brooks, Community Moderator Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.
    Terms & ConditionsAccessibility statement