Benefits of setting up HDF instead of plain NiFi?

[#1]

We are running NiFi instances "next to" Hortonworks HDP. This means we manually installed NiFi on our cluster nodes and let them connect to our ZooKeeper.

What are the benefits of installing HDF next to HDP on our nodes when we

1. only use NiFi and no other components such as MiNiFi or Kafka at the moment
2. we use NiFi to process incoming files (files the get stored in a specific folder), transform these files, put them into HDFS, create logs about the transformation process, finish data flow
3. after the transformed files are stored in HDFS they get analyzed in some way

Should we consider using HDF instead of "plain" NiFi?

Is it okay to run NiFi instances next to the HDP, without using HDF? Which problems can we run into?

Thanks!

[#2 · Accepted solution]

Hi Jody,

The benefits are integration with Ranger for authorizations management ; configuration management and deployment with Ambari (if you have a NiFi cluster, no need to manually update the configuration files on all the nodes) ; integrated monitoring with AMS/Grafana.

Please note that with Ambari 2.5.1.0 / HDP 2.6.1, it is possible to add the HDF management pack in the same Ambari to have both HDP and HDF components managed by the same Ambari and using the same common components (Ambari Infra, Zookeeper, Ranger, AMS, etc).

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/index.html#bk_installing-hdf-on-hdp

Hope this helps.

[#3]

Does it mean, if i have 3 node hdp, i can add nifi with the same node?

[#4]

Hi @mel mendoza,

NiFi is I/O intensive and is requiring dedicated disks. It is not recommended to use the NiFi nodes for something else (at least for a production environment). Besides, for very high rates workflows it is also highly recommended to have dedicated nodes (and disks) for Zookeeper nodes.

[#5]

This would be the perfect solution for our cluster setup.

How stable is this?

[#6]

In regards to NiFi inside HDF 3.0, this is very stable and brings a lot of new great features. Regarding the new components coming with HDF 3.0 (SAM and Schema Registry), you might want to wait HDF3.1 to start using the components in a production environment (of course it depends what are your requirements).