Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Benefits of setting up HDF instead of plain NiFi?

Solved Go to solution

Benefits of setting up HDF instead of plain NiFi?

New Contributor

We are running NiFi instances "next to" Hortonworks HDP. This means we manually installed NiFi on our cluster nodes and let them connect to our ZooKeeper.

What are the benefits of installing HDF next to HDP on our nodes when we

  1. only use NiFi and no other components such as MiNiFi or Kafka at the moment
  2. we use NiFi to process incoming files (files the get stored in a specific folder), transform these files, put them into HDFS, create logs about the transformation process, finish data flow
  3. after the transformed files are stored in HDFS they get analyzed in some way

Should we consider using HDF instead of "plain" NiFi?

Is it okay to run NiFi instances next to the HDP, without using HDF? Which problems can we run into?

Thanks!

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Benefits of setting up HDF instead of plain NiFi?

Hi Jody,

The benefits are integration with Ranger for authorizations management ; configuration management and deployment with Ambari (if you have a NiFi cluster, no need to manually update the configuration files on all the nodes) ; integrated monitoring with AMS/Grafana.

Please note that with Ambari 2.5.1.0 / HDP 2.6.1, it is possible to add the HDF management pack in the same Ambari to have both HDP and HDF components managed by the same Ambari and using the same common components (Ambari Infra, Zookeeper, Ranger, AMS, etc).

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/index.html#bk_installing-hdf-on-hdp

Hope this helps.

5 REPLIES 5

Re: Benefits of setting up HDF instead of plain NiFi?

Hi Jody,

The benefits are integration with Ranger for authorizations management ; configuration management and deployment with Ambari (if you have a NiFi cluster, no need to manually update the configuration files on all the nodes) ; integrated monitoring with AMS/Grafana.

Please note that with Ambari 2.5.1.0 / HDP 2.6.1, it is possible to add the HDF management pack in the same Ambari to have both HDP and HDF components managed by the same Ambari and using the same common components (Ambari Infra, Zookeeper, Ranger, AMS, etc).

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.0/index.html#bk_installing-hdf-on-hdp

Hope this helps.

Re: Benefits of setting up HDF instead of plain NiFi?

Contributor

Does it mean, if i have 3 node hdp, i can add nifi with the same node?

Re: Benefits of setting up HDF instead of plain NiFi?

Hi @mel mendoza,

NiFi is I/O intensive and is requiring dedicated disks. It is not recommended to use the NiFi nodes for something else (at least for a production environment). Besides, for very high rates workflows it is also highly recommended to have dedicated nodes (and disks) for Zookeeper nodes.

Re: Benefits of setting up HDF instead of plain NiFi?

New Contributor

This would be the perfect solution for our cluster setup.

How stable is this?

Re: Benefits of setting up HDF instead of plain NiFi?

In regards to NiFi inside HDF 3.0, this is very stable and brings a lot of new great features. Regarding the new components coming with HDF 3.0 (SAM and Schema Registry), you might want to wait HDF3.1 to start using the components in a production environment (of course it depends what are your requirements).

Don't have an account?
Coming from Hortonworks? Activate your account here