Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Adding NiFi Server to HDP 2.5 Sandbox

avatar
Master Guru

When I try to add the NiFi Service to the HDP 2.5 sandbox using Ambari, it fails. I have tried a few times on clean unchanged sandboxes and it always fails. There are a lot of warnings on configurations.

Is there a specific set of configurations needed to make this work?

1 ACCEPTED SOLUTION

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
9 REPLIES 9

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Master Guru

I will download it myself, I know that works, thanks. I usually do 8090. I just saw it in the list when I was working on the 2.5 sandbox and thought that would be easier.

5622-nifi.png

avatar
Master Guru

That is really interesting that NiFi is showing up in that list. I was assuming you had added the unofficial NiFi service definition, but I had no idea it was in the sandbox by default. I would be curious to know where that is coming from.

avatar
Master Guru

It is the HDP 2.5 Tech Preview Sandbox. Maybe experimental.

avatar
Guru

A similar question was posted with an answer similar to Bryan's, but some workarounds were suggested as well: https://community.hortonworks.com/questions/43459/unable-to-install-nifi-using-ambari-on-hdp24-sandb...

avatar

+@rmolina

@Timothy Spann: What error does it fail with? Sandbox team had included the demo Nifi service definition (https://github.com/abajwa-hw/ambari-nifi-service) on 2.4 sandbox to make it easy to install Nifi for the tutorials

In 2.4 the service definition was under below: /var/lib/ambari-server/resources/stacks/HDP/2.4/services/NIFI/

Can you check if NIFI dir is present under both /var/lib/ambari-server/resources/stacks/HDP/2.4/services/ and /var/lib/ambari-server/resources/stacks/HDP/2.5/services directories?

If so, remove the 2.4 version and restart ambari-server before installing

avatar
Master Guru
[root@sandbox kafka]# ls -lt  /var/lib/ambari-server/resources/stacks/HDP/2.4/services/
total 100
-rw-r--r-- 1 root root 1216 2016-06-25 16:17 stack_advisor.pyc
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 SQOOP
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 STORM
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 TEZ
drwxr-xr-x 3 root root 4096 2016-06-23 23:25 YARN
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 ZOOKEEPER
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 RANGER
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 RANGER_KMS
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 SLIDER
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 SPARK
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 KNOX
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 MAHOUT
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 OOZIE
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 PIG
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 FALCON
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 FLUME
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 HBASE
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 HDFS
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 HIVE
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 KAFKA
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 KERBEROS
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 ACCUMULO
drwxr-xr-x 2 root root 4096 2016-06-23 23:25 ATLAS
-rwxr-xr-x 1 root root 1190 2016-06-23 17:47 stack_advisor.pyo
-rwxr-xr-x 1 root root  850 2016-06-23 17:41 stack_advisor.py
[root@sandbox kafka]# ls -lt /var/lib/ambari-server/resources/stacks/HDP/2.5/services
total 300
-rw-r--r-- 1 root root 51905 2016-06-26 21:45 stack_advisor.pyc
-rwxr-xr-x 1 root root 95429 2016-06-26 21:45 stack_advisor.py
drwxr-xr-x 7 root root  4096 2016-06-24 00:35 NIFI
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 YARN
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 ZEPPELIN
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 ZOOKEEPER
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 STORM
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 TEZ
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 SPARK
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 SPARK2
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 SQOOP
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 RANGER
drwxr-xr-x 4 root root  4096 2016-06-23 23:25 RANGER_KMS
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 SLIDER
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 OOZIE
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 PIG
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 KNOX
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 MAHOUT
drwxr-xr-x 5 root root  4096 2016-06-23 23:25 HIVE
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 KAFKA
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 KERBEROS
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 HBASE
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 HDFS
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 FALCON
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 FLUME
drwxr-xr-x 3 root root  4096 2016-06-23 23:25 ATLAS
drwxr-xr-x 2 root root  4096 2016-06-23 23:25 ACCUMULO
-rwxr-xr-x 1 root root 51651 2016-06-23 17:47 stack_advisor.pyo
[root@sandbox kafka]	

It won't install and states install failed. It leaves things in a bad state and changes a bunch of YARN and HDFS configs.

avatar

@Timothy Spann Ok finally got a chance to try this out. Couple of things:

1. You can remove failed installs easily in Ambari 2.4 via the UI by clicking on Nifi > Service Actions > Delete

2. Nifi is not causing the popup about HDFS/YARN configs. If you try to add ANY service it will give you the same warnings. Why? Because on sandbox they have setup their own configs to get the VM to perform on single node env, and Ambari is warning about those. I think it should be safe to ignore those.

3. Installing Nifi via Ambari was failing because of some Ambari API changes in 2.4. So any service definition written prior to 2.4 will probably fail without making some updates. I have now updated the code on the github now so should work on both Ambari 2.4, as well as earlier versions as well. So to try it out, once you have imported the 2.5 sandbox, just run the below commands and proceed with the install as usual (via 'Add service' wizard):

VERSION=`hdp-select status hadoop-client | sed 's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'`
rm -rf /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/NIFI  

sudo git clone https://github.com/abajwa-hw/ambari-nifi-service.git   /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/NIFI

service ambari restart 

Once installed, you can access Nifi UI on port 9090 (there is already a port forwarding rule on the VM). Also note that by default you will not see any metrics for Nifi (because Ambari Metrics is down)

Screenshots

5691-screen-shot-2016-07-09-at-95706-pm.png

5687-screen-shot-2016-07-09-at-94509-pm.png

5688-screen-shot-2016-07-09-at-95410-pm.png

5690-screen-shot-2016-07-09-at-95222-pm.png

@rmolina fyi...when you build your next sandbox, you should get the updated Nifi service definition automatically. So with future 2.5 sandboxes, the above manual workaround will no longer be required

avatar
Contributor

I've had zero success trying to get NIFI installed via Ambari. When it throws the recommended settings screen, it won't let me proceed. Additionally, I tried the above steps to rm the files, I consistently get

cannot remove `/var/lib/ambari-server/resources/stacks/HDP/2.5/services/NIFI/package/scripts': Invalid argument

I seem to get the invalid argument "feature" whenever I try and remove stuff from this docker package. Any guidance would be appreciated.

Thanks,