question Re: HDF 2.0 - with Storm and Kafka, How is this embedded in YARN ? in Support Questions

HDF 2.0 - with Storm and Kafka, How is this embedded in YARN ?

gerald_both — Mon, 19 Sep 2016 15:42:31 GMT

On Hortonworks Data Flow 2.0. My understanding is that Storm and Kafka are embedded now (potentially to provide e.g. more and faster throughput). My understanding is also that Apache NiFi (or HDF 1.0) does not run on YARN. Now with HDF 2.0 which has Apache Kafka and Storm as components as well. Is HDF 2.0 then deployed on Yarn ? Or is it still a standalone cluster and if yes where is then Apache Kafka and Storm running ? On another HDP cluster ?

kind regards, Gerald

Re: HDF 2.0 - with Storm and Kafka, How is this embedded in YARN ?

sgowda — Mon, 19 Sep 2016 15:45:34 GMT

Hello @Gerald Both,

No, even in HDF 2.0 storm and Kafka runs as a standalone cluster and are not deployed on yarn.

Thanks

Santhosh

Re: HDF 2.0 - with Storm and Kafka, How is this embedded in YARN ?

gerald_both — Mon, 19 Sep 2016 16:30:56 GMT

thank you. so would it be like 3 clusters then ? hdf on one, storm on another and kafka on the third ? are there plans to integrate HDF on yarn ?

Re: HDF 2.0 - with Storm and Kafka, How is this embedded in YARN ?

sgowda — Mon, 19 Sep 2016 20:56:46 GMT

With HDF 2.0, Ambari supports Nifi management, so all ( nifi , storm , kafka ) of these can be part of single cluster. Regarding HDF on yarn -- dont think thats there on the radar.

Re: HDF 2.0 - with Storm and Kafka, How is this embedded in YARN ?

TimothySpann — Mon, 19 Sep 2016 21:41:01 GMT

No YARN in HDF, that's why it's a separate cluster. To get YARN you need all the other Hadoop pieces and that makes for a much larger node.

Re: HDF 2.0 - with Storm and Kafka, How is this embedded in YARN ?

gerald_both — Wed, 21 Sep 2016 14:42:39 GMT

thank you all