Created 07-10-2017 06:56 AM
Hello!
Found interesting information:
Starting from HDP 3.0 release Kafka will be moved from HDP. So it will be supported only in HDF
Our case:
Cluster1:
We have a Hadoop cluster on HDP.
Cluster2:
Also we have seperated N nodes for Kafka cluster.
But on this seperate cluster we install HDP with: Kafka/Yarn/Hadoop/Spark2/Oozie
Second cluster used as DataBus and Streaming data processing cluster.
Yarn&HDFS installed only for need of Spark2. We does not store any data in this HDFS
Oozie is used for job configurations to start/stop/monitor jobs.
The reason why we run spark2 streaming jobs not on the Cluster1 - is resources and stability. Cluster2 (Kafka cluster) have a lot of free momory and unused CPUs. Cluster1 have no free resources and always used for 100%.
In our case streaming jobs should be as stable as possible, but we does not want to limit resources of main cluster (Cluster 1).
Up to version 2.6 everything was OK.
But if Kafka will be moved I can't understand how to install HDF (kafka) & HDP (HDFS+Yarn+Spark2+Oozie) on the same nodes.
The solutions:
1#
- install HDP 3 (HDFS+Yarn+Spark2+Oozie)
- install standalone Kafka, or another platform (e.g. Confluent Platform), but we would prefer to use Hortonworks stack
2#
- install HDF 3 (Kafka)
- manual HDFS+Yarn installation (very complicated task)
3#
- install HDF 3 (Kafka)
- run Spark streming jobs not on Yarh cluster, but on Spark own managed Cluster on the same node as Kafka.
This is very complicated tasks because we should reconfigure and test more than 30 streaming jobs and rewrite our monitoring engine.
Up to now we would chose solution #1. But It would be great if we could combine power of HDP and HDF on the same nodes.
Created 07-10-2017 02:21 PM
The latest version of Ambari supports installing and managing both HDP and HDF. If you upgrade Ambari on the HDP cluster, you could install the HDF management pack and then install Kafka via Ambari that way.
Created 07-10-2017 02:43 PM
Thanks a lot for the help!
Created 08-25-2017 08:22 AM
@Michael Young, Will ambari 2.5.0.3 support both HDP and HDF (as management pack) or ambari 2.5.1.0 is required?
Created 04-09-2018 05:03 PM
@Michael Young , I am using Ambai 2.6.0.0. Will this ambari support the Nifi Demo by installing Mpack? I am trying to make it happen but failed. Would you please help me How can I run nifi from HDP sandbox? Thank you.