best practices - Integrate Nifi with existing HDP

[#1]

We have an existing HDP cluster with the typical infrastructure setup: hive/spark/oozie/ranger (all kerberized). What is the best way to integrate a nifi cluster to this cluster? Keeping in mind upgrading and easy existing integration with security etc.

[#2 · Accepted solution]

@Sonny Heer

The HDP stack does not include NiFi. HDF is an entirely different software stack managed by its own Ambari and is installed on its own hardware. There are some post out there that will walk you through manually adding the NiFi service to an existing HDP stack; however, doing so breaks the upgradeability of your HDP stack.

Here is an example of that:

https://github.com/abajwa-hw/ambari-nifi-service

*** Pay special attention to the "Limitations" noted on the above link.

At some point in the future there will be an Ambari platform that can be used to install both HDP and HDF stack services that will still support the upgradeability, but that is not here yet and I don't know hen that will happen.

Thank you,

Matt

[#3]

Thanks @Matt Clarke

I understand HDF is separate. I'm looking for the best approach to use existing HDP with HDF. I'll look into ambari-nifi-service again - i didn't pursue this service because of the disclaimer about it being for demo purposes. Anyone using this service - is it up to date with 2.4? Assuming i don't care about upgradability at the moment...

Prefer to add nifi cluster to existing HDP kerberized cluster. any ideas/posts/links on how to best do that would be helpful. Is the best approach to setup a whole new ambari system? I would like to avoid doing this. Thanks