Installing HDF (NiFi)

[#1]

Last year I setup a HDP 2.4 cluster with four nodes and I installed HDF (Nifi). At that time I simple put NiFi on node1 of the HDP cluster, installing it on the same server as Ambari. Everything worked fine and it was simple.

I have now installed HDP2.5 in a new four node cluster and I have read that you can no longer install HDF on the same cluster as HDP. The HDF install has it's own instance of Ambari and requires a separate cluster.

Here are my questions:

(1) Must the HDF instance be on a "cluster" or can I simply have a single server for the HDF instance? ...I am not certain what would have changed from a year ago that would require a full cluster for HDF?

(2) With the new server/cluster for the HDF, does that instance of NiFi then connect and write data back to the HDP cluster?

Sorry if these are obvious questions, but I was unclear and trying to gain an understanding of exactly what is needed.

Thank you, -Marc

[#2 · Accepted solution]

@marksf

With HDP 2.4 what version of HDF did you install? Yes we cannot have HDP and HDF managed by same Ambari yet.

You can install NiFi only tar on your HDP 2.5 but it wont be managed by HDP Ambari.

HDF can be a standalone instance or a cluster as well. It can connect to HDP cluster using the client libraries available in NiFi lib (nars).

The HDF cluster can have NiFi, Zookeeper, Kafka, Storm, Ranger, Ambari Metrics, Ambari Infra and Log search. However you can choose the services you would like to have similar to HDP and install only NiFi and zookeeper to start with.

Hope this helps.

[#3]

Thank you for the thorough answer. I am good to give this a try now.

I cannot recall the previous HDF version I installed, but I think it was based on Nifi 0.6. I was back in, roughly, April of 2016.

Thank you again!