HDF 2.0 - with Storm and Kafka, How is this embedded in YARN ?

[#1]

On Hortonworks Data Flow 2.0. My understanding is that Storm and Kafka are embedded now (potentially to provide e.g. more and faster throughput). My understanding is also that Apache NiFi (or HDF 1.0) does not run on YARN. Now with HDF 2.0 which has Apache Kafka and Storm as components as well. Is HDF 2.0 then deployed on Yarn ? Or is it still a standalone cluster and if yes where is then Apache Kafka and Storm running ? On another HDP cluster ?

kind regards, Gerald

[#2 · Accepted solution]

Hello @Gerald Both,

No, even in HDF 2.0 storm and Kafka runs as a standalone cluster and are not deployed on yarn.

Thanks

Santhosh

[#3]

thank you. so would it be like 3 clusters then ? hdf on one, storm on another and kafka on the third ? are there plans to integrate HDF on yarn ?

[#4]

With HDF 2.0, Ambari supports Nifi management, so all ( nifi , storm , kafka ) of these can be part of single cluster. Regarding HDF on yarn -- dont think thats there on the radar.

[#5]

No YARN in HDF, that's why it's a separate cluster. To get YARN you need all the other Hadoop pieces and that makes for a much larger node.

[#6]

thank you all