Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Benefits of setting up HDF instead of plain NiFi?

avatar
Explorer

We are running NiFi instances "next to" Hortonworks HDP. This means we manually installed NiFi on our cluster nodes and let them connect to our ZooKeeper.

What are the benefits of installing HDF next to HDP on our nodes when we

  1. only use NiFi and no other components such as MiNiFi or Kafka at the moment
  2. we use NiFi to process incoming files (files the get stored in a specific folder), transform these files, put them into HDFS, create logs about the transformation process, finish data flow
  3. after the transformed files are stored in HDFS they get analyzed in some way

Should we consider using HDF instead of "plain" NiFi?

Is it okay to run NiFi instances next to the HDP, without using HDF? Which problems can we run into?

Thanks!

1 ACCEPTED SOLUTION

avatar

Hi Jody,

The benefits are integration with Ranger for authorizations management ; configuration management and deployment with Ambari (if you have a NiFi cluster, no need to manually update the configuration files on all the nodes) ; integrated monitoring with AMS/Grafana.

Please note that with Ambari 2.5.1.0 / HDP 2.6.1, it is possible to add the HDF management pack in the same Ambari to have both HDP and HDF components managed by the same Ambari and using the same common components (Ambari Infra, Zookeeper, Ranger, AMS, etc).

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/index.html#bk_installing-hdf-on-hdp

Hope this helps.

View solution in original post

5 REPLIES 5

avatar

Hi Jody,

The benefits are integration with Ranger for authorizations management ; configuration management and deployment with Ambari (if you have a NiFi cluster, no need to manually update the configuration files on all the nodes) ; integrated monitoring with AMS/Grafana.

Please note that with Ambari 2.5.1.0 / HDP 2.6.1, it is possible to add the HDF management pack in the same Ambari to have both HDP and HDF components managed by the same Ambari and using the same common components (Ambari Infra, Zookeeper, Ranger, AMS, etc).

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/index.html#bk_installing-hdf-on-hdp

Hope this helps.

avatar
Contributor

Does it mean, if i have 3 node hdp, i can add nifi with the same node?

avatar

Hi @mel mendoza,

NiFi is I/O intensive and is requiring dedicated disks. It is not recommended to use the NiFi nodes for something else (at least for a production environment). Besides, for very high rates workflows it is also highly recommended to have dedicated nodes (and disks) for Zookeeper nodes.

avatar
Explorer

This would be the perfect solution for our cluster setup.

How stable is this?

avatar

In regards to NiFi inside HDF 3.0, this is very stable and brings a lot of new great features. Regarding the new components coming with HDF 3.0 (SAM and Schema Registry), you might want to wait HDF3.1 to start using the components in a production environment (of course it depends what are your requirements).