How to install NiFi on Ambari on Azure HDInsight Spark cluster

[#1]

Hi

I am trying to get NiFi on Ambari on an Azure HDInsight cluster.

I have tried following most of the links in the forums on installing NiFi on Ambari but cannot find anything that would specifically touch on the HDI on Azure.

It seems like the Add Service option on Ambari is contained by the service that Azure is providing, i have not found anything saying that this is case but the Add Service option is not there.

Another option that i have not tried is to install the NiFi via Script

https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-hadoop-customize-cluster-linux#example-sc...

or

https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apps-install-custom-applications

Anyone got NiFi to work in Ambari on HDI on Azure?

Anyone have a script for the NiFi installation that can be used according above steps?

Br

Michael

[#2 · Accepted solution]

Hi @Michael Jonsson

NiFi is not packaged in HDI this is why you can not find it in Ambari from Add Service. You can install an HDP + HDF cluster on Azure in IaaS mode to have both platforms. You can also provision Azure VMs and install Ambari + HDF only, and use it to with HDI. This way you have two separate clusters HDF and HDI. You can also use Cloudbreak for easier installation.

Theoretically, you should be able to manually install NiFi on HDI nodes but this won't be supported neither manager by Ambari (no monitoring, configuration, upgrade, etc ..). So this may make sense for testing/POC. I've never tried it though.

[#3]

Hi @Abdelkrim Hadjidj

Thanks for your response and tips. One of the reasons i would like to have it in the HDI is to have one place of management (also for governance).

If i look at this guide, i get the notion that NiFi can be in the installed in Ambari.

https://community.hortonworks.com/content/kbentry/51517/hdf-on-hdi-nifi.html

But as you say the question if it is supported etc remains, was hoping someone had done this and could enlighten on the topic.

[#4]

Hi @Michael Jonsson

The link you refer to uses a custom Ambari Service for NiFi. It's not official neither supported: https://github.com/abajwa-hw/ambari-nifi-service

As you can see, the post is old and the service is no more updated by its author (see deprication note on the Github page).

[#5]

Hi @Abdelkrim Hadjidj

Thanks for the update, i also looked into this link (read the last post) and tried those steps.

https://community.hortonworks.com/questions/64996/installing-nifi-using-ambari.html

The add service is not present in the Azure provided HDI though and i do not know how to add it, seems like it should be possible to achieve to have NiFi in Ambari, not sure on the Azure HDI but keen on if somebody got it working.

[#6]

Hi @Michael Jonsson

Again, NiFi is not available on HDI and you can not install it from Ambari in HDI.

Starting from HDF 3 it's possible to deploy HDF component on HDP cluster using management packs but this is for HDP not HDI : https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.0.1.1/bk_installing-hdf-on-hdp/content/ch_insta...

The link you provided is also deprecated, lot of things changed since then 🙂

[#7]

Ok, thanks again for clarification.

Building an understanding on HDI, HDF, and HDP here 🙂