Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Step by Step process to create a clustered NiFi configuration?

Highlighted

Step by Step process to create a clustered NiFi configuration?

Contributor

I have been digging through all of the sections of the NiFi Admin guide, looking for NiFi Best Practices Documents and anything else I can get my hands on to read, trying to piece together how to configure a secured, and a non-secured, NiFi clustered configuration.

It would a huge time saver if someone from Hortonworks would write a 'step by step' article defining: each properties file to edit and what specific parameters to set for the Slave Nodes and what to set for the NiFi Cluster Manager (NCM).

Thank you to anyone willing to do this from Hortonworks.

5 REPLIES 5

Re: Step by Step process to create a clustered NiFi configuration?

Matt Clarke from the NiFi support team has written an excellent article on this.

Re: Step by Step process to create a clustered NiFi configuration?

Re: Step by Step process to create a clustered NiFi configuration?

Yes, particularly the section Clustering Configuration

Re: Step by Step process to create a clustered NiFi configuration?

Contributor

Hi Andy and Sagar,

Thank you for your replies.

Andy - I had already read Matt's article - yes, it is an awesome article, 'if' you already know, or have worked with NiFi and his article deals with "migration" from a Standalone config, not specific details regarding setting up a brand new clustered configuration.

Sagar - I have already read that section of the manual many times, if I hadn't read it, your suggestion to read it was a great idea.

Please know, I am making these this request in the spirit of making this process easier for others in the future; I appreciate ALL of the efforts of people like Matt, and everyone else, who replies to this website or writes articles.

Room for enhancement in the manuals: When the NiFi Admin manual, in chapter 13 - System Parameters, shows a chart of 'Parameter names' and the 'description' values for those parameters say phrases like: "It is blank by default." That is not a description, that tells us nothing about the parameter, it's purpose, what other parameters it could affect, or anything useful. It just says it's 'blank'. So... Do I need to enter something for this parm, or do I just leave it 'blank' and hope that by leaving it 'blank' doesn't hose the configuration? Also more actual examples would be a great help in the manuals too.

Bottomline:

If you don't know NiFi, or if you are a first time Admin of NiFi, Matt's article and what is in the Manuals, makes large assumptions about the background of and knowledge of, who are the readers of these documents.

Repeating, what is needed is : A 'step by step' article or rewrite the manual, specifically defining the setup needed for a NEW NIFI CLUSTERED environment:

1) a specfic List of each Properties / Configuration files to edit

and

2) What specific parameters to set for the Slave Nodes in each of the properties / configuration files

and

3) What specific parameters to set for the NiFi Cluster Manager (NCM).

Re: Step by Step process to create a clustered NiFi configuration?

Take a look at https://community.hortonworks.com/repos/8878/nifihdf-docker-environment-with-clustering-support.html

There is a run.sh script file which can give you an idea of exact steps performed for each node role (e.g. a jump host, NCM, worker)

Don't have an account?
Coming from Hortonworks? Activate your account here