Adding NiFi Server to HDP 2.5 Sandbox

[#1]

When I try to add the NiFi Service to the HDP 2.5 sandbox using Ambari, it fails. I have tried a few times on clean unchanged sandboxes and it always fails. There are a lot of warnings on configurations.

Is there a specific set of configurations needed to make this work?

[#2 · Accepted solution]

There is no official NiFi service for Ambari yet. It is recommended to download the NiFi tar/zip on the sandbox, extract it and change the web port in nifi.properties to something besides 8080, and forward that port through the VM.

[#3]

I will download it myself, I know that works, thanks. I usually do 8090. I just saw it in the list when I was working on the 2.5 sandbox and thought that would be easier.

[#4]

That is really interesting that NiFi is showing up in that list. I was assuming you had added the unofficial NiFi service definition, but I had no idea it was in the sandbox by default. I would be curious to know where that is coming from.

[#5]

It is the HDP 2.5 Tech Preview Sandbox. Maybe experimental.

[#6]

A similar question was posted with an answer similar to Bryan's, but some workarounds were suggested as well: https://community.hortonworks.com/questions/43459/unable-to-install-nifi-using-ambari-on-hdp24-sandb...

[#7]

+@rmolina

@Timothy Spann: What error does it fail with? Sandbox team had included the demo Nifi service definition (https://github.com/abajwa-hw/ambari-nifi-service) on 2.4 sandbox to make it easy to install Nifi for the tutorials

In 2.4 the service definition was under below: /var/lib/ambari-server/resources/stacks/HDP/2.4/services/NIFI/

Can you check if NIFI dir is present under both /var/lib/ambari-server/resources/stacks/HDP/2.4/services/ and /var/lib/ambari-server/resources/stacks/HDP/2.5/services directories?

If so, remove the 2.4 version and restart ambari-server before installing

[#8]

[root@sandbox kafka]# ls -lt  /var/lib/ambari-server/resources/stacks/HDP/2.4/services/
total 100
-rw-r--r-- 1 root root 1216 2016-06-25 16:17 stack_advisor.pyc
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 SQOOP
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 STORM
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 TEZ
drwxr-xr-x 3 root root 4096 2016-06-23 23:25 YARN
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 ZOOKEEPER
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 RANGER
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 RANGER_KMS
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 SLIDER
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 SPARK
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 KNOX
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 MAHOUT
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 OOZIE
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 PIG
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 FALCON
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 FLUME
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 HBASE
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 HDFS
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 HIVE
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 KAFKA
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 KERBEROS
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 ACCUMULO
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 ATLAS
-rwxr-xr-x 1 root root 1190 2016-06-23 17:47 stack_advisor.pyo
-rwxr-xr-x 1 root root  850 2016-06-23 17:41 stack_advisor.py
[root@sandbox kafka]# ls -lt /var/lib/ambari-server/resources/stacks/HDP/2.5/services
total 300
-rw-r--r-- 1 root root 51905 2016-06-26 21:45 stack_advisor.pyc
-rwxr-xr-x 1 root root 95429 2016-06-26 21:45 stack_advisor.py
drwxr-xr-x 7 root root  4096 2016-06-24 00:35 NIFI
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 YARN
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 ZEPPELIN
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 ZOOKEEPER
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 STORM
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 TEZ
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 SPARK
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 SPARK2
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 SQOOP
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 RANGER
drwxr-xr-x 4 root root  4096 2016-06-23 23:25 RANGER_KMS
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 SLIDER
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 OOZIE
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 PIG
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 KNOX
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 MAHOUT
drwxr-xr-x 5 root root  4096 2016-06-23 23:25 HIVE
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 KAFKA
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 KERBEROS
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 HBASE
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 HDFS
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 FALCON
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 FLUME
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 ATLAS
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 ACCUMULO
-rwxr-xr-x 1 root root 51651 2016-06-23 17:47 stack_advisor.pyo
[root@sandbox kafka]	

It won't install and states install failed. It leaves things in a bad state and changes a bunch of YARN and HDFS configs.

[#9]

@Timothy Spann Ok finally got a chance to try this out. Couple of things:

1. You can remove failed installs easily in Ambari 2.4 via the UI by clicking on Nifi > Service Actions > Delete

2. Nifi is not causing the popup about HDFS/YARN configs. If you try to add ANY service it will give you the same warnings. Why? Because on sandbox they have setup their own configs to get the VM to perform on single node env, and Ambari is warning about those. I think it should be safe to ignore those.

3. Installing Nifi via Ambari was failing because of some Ambari API changes in 2.4. So any service definition written prior to 2.4 will probably fail without making some updates. I have now updated the code on the github now so should work on both Ambari 2.4, as well as earlier versions as well. So to try it out, once you have imported the 2.5 sandbox, just run the below commands and proceed with the install as usual (via 'Add service' wizard):

VERSION=`hdp-select status hadoop-client | sed 's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'`
rm -rf /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/NIFI  

sudo git clone https://github.com/abajwa-hw/ambari-nifi-service.git   /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/NIFI

service ambari restart 

Once installed, you can access Nifi UI on port 9090 (there is already a port forwarding rule on the VM). Also note that by default you will not see any metrics for Nifi (because Ambari Metrics is down)

Screenshots

@rmolina fyi...when you build your next sandbox, you should get the updated Nifi service definition automatically. So with future 2.5 sandboxes, the above manual workaround will no longer be required

[#10]

I've had zero success trying to get NIFI installed via Ambari. When it throws the recommended settings screen, it won't let me proceed. Additionally, I tried the above steps to rm the files, I consistently get

cannot remove `/var/lib/ambari-server/resources/stacks/HDP/2.5/services/NIFI/package/scripts': Invalid argument

I seem to get the invalid argument "feature" whenever I try and remove stuff from this docker package. Any guidance would be appreciated.

Thanks,