Support Questions
Find answers, ask questions, and share your expertise

Installing Nifi with cloudera

Explorer

Hi , I am trying to install apache nifi on cloudera and i want to manage nifi with cloudera manager. I am able to build a parcel and distribute activate on all the nodes. After the csd creation when i restart the cloudera server the nifi service never comes up in the add a service wizard.I pretty much followed this "https://github.com/prateek/nifi-parcel". One thing is we are not using HDFS and instead we are using isilon. Any help in resolving this issue would greatly help. If some one could post thier .sdl and contol.sh would also be great.

1 ACCEPTED SOLUTION

Super Collaborator

Since the question was asked, the situation has changed. As soon as Hortonworks and Cloudera merged, NiFi became supported by Cloudera.

 

Shortly after the integrations with CDH were also completed, so that NiFi is now a fully supported and integrated component.

 

Please look into the documentation for the latest info at any time, but in general Cloudera Manager is now able to install NiFi.


- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.

View solution in original post

15 REPLIES 15

Champion
Disclaimer: I haven't done this at all.

Did you change the service.sdl so that HDFS wasn't a requirement or so that Isilon is one?

I don't think that matters for what you are experiencing as the dependencies and everything else in service.sdl comes into play when you go through the Add a Service wizard in CM. The parcel.json should define what comes in the parcel. I would think it being listed in the packages/components is what is need to have it show up as a service that can be added.

Did it create the NIFI user? Did it unpack the parcel on all nodes? Those are a couple of items that may point to whether the parcel deployed correctly.

"users": {
"nifi": {
"longname" : "NiFi",
"home" : "/var/lib/nifi",
"shell" : "/bin/bash",
"extra_groups": []
}

Explorer

My issue was with Add a service Wizard ,Nifi not bieng displayed. I did check what you mentioned and everything seems to be in place. The parcel is fine (downloaded,ditributed,activated) as expected on all the nodes through cloudera manager.It did unpack the parcel on all the nodes.

Rising Star

From the description I would like to say that the csd may not be installed on the CM in question.

A quick way to check this is :

http://CLOUDERA_MANAGER:7180/csd/refresh. Check if you can see the CSD present and is installed.

If not you will need to install the CSD and after that you will be able to see the service.

Explorer

Disclaimer: First time writing a csd

 

I overlooked the Local descriptor repository path . It is pointing to some other directory. I made it point to the csd and restarted the server. When i do the refresh it gives the following error.

invalidCsds":{"/opt/app/cloudera/csd/NIFI-1.0":["Failed to load CSD from /opt/app/cloudera/csd/NIFI-1.0, Error: CSD bundle data directory or Jar file doesn't exist"],"/opt/app/cloudera/csd/NIFI-1.0.jar":["Failed to load CSD from /opt/app/cloudera/csd/NIFI-1.0.jar, Error: CSD bundle data directory or Jar file doesn't exist"]},"repoEnabled":true},"message":"OK"}

Any Pointers would help me to overcome this issue

Rising Star

I am not sure how you have deployed the csd and the JAR file in the cluster.

Usually the location is somewhat similar to /opt/cloduera/csd/. If you put the JAR file out there and when you run the /csd/refresh link you should see the "NIFI" csd in the list.

After that, you will need to install the csd at /csd/install.

 

Do note you need to put the jar only on the CM node and dont need to install it on all the nodes.

 

Doc link : http://github.mtv.cloudera.com/Starship/cm_ext/wiki/CSD-Developer-Tricks-and-Tools

 

Hope it helps.

Explorer

I just went through the  developer Tricks and tools, those API's did help me, I am able to succesfully install and start Nifi from the cloudera manager and see the web UI as well ,but i have deployed the csd again in the default location. For some reason I am getting error while starting nifi by the cloudera manager if it is in my custom location, have to figure it out still.

Explorer

Hello,

Did  you sort it out your nifi on CM? I am looking for a solution on how to install NiFi on Cloudera cluster and using it from CM but have not seen a working walkthrough. Can you share your procedure? Thank you very much in advance.

Explorer

Hey sorry for the late reply. I moved out from the company. I finally ended up using stand alone nifi cluster instead of using it from cloudera manager.

 

However i am able to deploy the nifi as a service in cluster mode from the cloudera manager.

 

My major concern at that time was not knowing how to change the properties in cloudera manager and make it reflect back to nifi.proprties. I did write some part of the csd which is very vauge and unfortunately i did not have a backup of the immature csd i have written.

 

I will try to get it if possible and  paste it here. 

 

Regards,

 PV 

Super Collaborator

Since the question was asked, the situation has changed. As soon as Hortonworks and Cloudera merged, NiFi became supported by Cloudera.

 

Shortly after the integrations with CDH were also completed, so that NiFi is now a fully supported and integrated component.

 

Please look into the documentation for the latest info at any time, but in general Cloudera Manager is now able to install NiFi.


- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.

New Contributor
Hi, could you please share more instructions how to install NiFi using Cloudera Manager? I am using Cloudera Express 5.14 Thanks!

Super Collaborator

Hi @michalr ,

 

Here is the documentation you can follow on how to install NiFi:

https://docs.hortonworks.com/HDPDocuments/CFM/CFM-1.0.0/installation/content/overview.html

 

Thanks,

Li

Li Wang, Technical Solution Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

Terms of Service

Community Guidelines

How to use the forum

New Contributor

@lwang 

Thank you for the link, after a couple of iterations this tutorial finally helped me. Thanks!


Thanks,

Michal

 

Super Collaborator
@michalr I believe you tried to reach out to me during my vacation, with a question about versions.

It seems you already found the solution, but for clarity: The current version of the docs https://docs.hortonworks.com/HDPDocuments/CFM/CFM-1.0.0/installation/content/cfm-system-requirements... refers to 5.5.6 and above. If you currently have 5.14.x that should be fine!

If in doubt, it is best to ensure you ask questions in public as I may not see messages personally in a reasonable amount of time!

- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.

Super Collaborator
Hello,

I also replied in the thread, but 5.14 is newer than 5.6 so that should be fine!

Kind regards,
Dennis

- Dennis Jaheruddin

If this answer helped, please mark it as 'solved' and/or if it is valuable for future readers please apply 'kudos'.

New Contributor

Hi Dennis,

 

As mentioned in the (edited) post, the solution suggested above finally worked for me. Thanks again for the help!

 

Regards,

Michal

; ;