Installing Nifi with cloudera

[#1]

Hi , I am trying to install apache nifi on cloudera and i want to manage nifi with cloudera manager. I am able to build a parcel and distribute activate on all the nodes. After the csd creation when i restart the cloudera server the nifi service never comes up in the add a service wizard.I pretty much followed this "https://github.com/prateek/nifi-parcel". One thing is we are not using HDFS and instead we are using isilon. Any help in resolving this issue would greatly help. If some one could post thier .sdl and contol.sh would also be great.

[#2]

Disclaimer: I haven't done this at all.

Did you change the service.sdl so that HDFS wasn't a requirement or so that Isilon is one?

I don't think that matters for what you are experiencing as the dependencies and everything else in service.sdl comes into play when you go through the Add a Service wizard in CM. The parcel.json should define what comes in the parcel. I would think it being listed in the packages/components is what is need to have it show up as a service that can be added.

Did it create the NIFI user? Did it unpack the parcel on all nodes? Those are a couple of items that may point to whether the parcel deployed correctly.

"users": {
"nifi": {
"longname" : "NiFi",
"home" : "/var/lib/nifi",
"shell" : "/bin/bash",
"extra_groups": []
}

[#3]

My issue was with Add a service Wizard ,Nifi not bieng displayed. I did check what you mentioned and everything seems to be in place. The parcel is fine (downloaded,ditributed,activated) as expected on all the nodes through cloudera manager.It did unpack the parcel on all the nodes.

[#4]

From the description I would like to say that the csd may not be installed on the CM in question.

A quick way to check this is :

http://CLOUDERA_MANAGER:7180/csd/refresh. Check if you can see the CSD present and is installed.

If not you will need to install the CSD and after that you will be able to see the service.

[#5]

Disclaimer: First time writing a csd

I overlooked the Local descriptor repository path . It is pointing to some other directory. I made it point to the csd and restarted the server. When i do the refresh it gives the following error.

invalidCsds":{"/opt/app/cloudera/csd/NIFI-1.0":["Failed to load CSD from /opt/app/cloudera/csd/NIFI-1.0, Error: CSD bundle data directory or Jar file doesn't exist"],"/opt/app/cloudera/csd/NIFI-1.0.jar":["Failed to load CSD from /opt/app/cloudera/csd/NIFI-1.0.jar, Error: CSD bundle data directory or Jar file doesn't exist"]},"repoEnabled":true},"message":"OK"}

Any Pointers would help me to overcome this issue

[#6]

I am not sure how you have deployed the csd and the JAR file in the cluster.

Usually the location is somewhat similar to /opt/cloduera/csd/. If you put the JAR file out there and when you run the /csd/refresh link you should see the "NIFI" csd in the list.

After that, you will need to install the csd at /csd/install.

Do note you need to put the jar only on the CM node and dont need to install it on all the nodes.

Doc link : http://github.mtv.cloudera.com/Starship/cm_ext/wiki/CSD-Developer-Tricks-and-Tools

Hope it helps.

[#7]

I just went through the  developer Tricks and tools, those API's did help me, I am able to succesfully install and start Nifi from the cloudera manager and see the web UI as well ,but i have deployed the csd again in the default location. For some reason I am getting error while starting nifi by the cloudera manager if it is in my custom location, have to figure it out still.

[#8]

Hello,

Did  you sort it out your nifi on CM? I am looking for a solution on how to install NiFi on Cloudera cluster and using it from CM but have not seen a working walkthrough. Can you share your procedure? Thank you very much in advance.

[#9]

Hey sorry for the late reply. I moved out from the company. I finally ended up using stand alone nifi cluster instead of using it from cloudera manager.

However i am able to deploy the nifi as a service in cluster mode from the cloudera manager.

My major concern at that time was not knowing how to change the properties in cloudera manager and make it reflect back to nifi.proprties. I did write some part of the csd which is very vauge and unfortunately i did not have a backup of the immature csd i have written.

I will try to get it if possible and  paste it here. 

Regards,

 PV 

[#10 · Accepted solution]

Since the question was asked, the situation has changed. As soon as Hortonworks and Cloudera merged, NiFi became supported by Cloudera.

Shortly after the integrations with CDH were also completed, so that NiFi is now a fully supported and integrated component.

Please look into the documentation for the latest info at any time, but in general Cloudera Manager is now able to install NiFi.