Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Executing apache NIFI through a shell script

Highlighted

Executing apache NIFI through a shell script

Explorer

I have designed a NIFI dataflow using 2 processors to take input from the local file system and load into Hadoop File system. I have the following queries with respect to the above:

  1. How can I execute the data flow inside a shell script instead of executing it via the GUI?
  2. Does apache NIFI provides guaranteed delivery. In case the job fails how the recovery is done? Do we need to restart the process from the start all over again? Say there are 100000 records in a file and the job fails after reading 20000 files.In this case once we restart does it execute the process from the beginning?

Regards,

I Roy

3 REPLIES 3
Highlighted

Re: Executing apache NIFI through a shell script

Expert Contributor
  • You can start a processor programatically using Nifi APIs - here is the link to the API documentation (specifically you will be interested in the updateProcessor API within the processors
PUT/controller/process-groups/{process-group-id}/processors/{id}    Updates a processor
  • Nifi keeps track of the processed messages, so when the process restarts it starts from where it left.. also you have an option to terminate the flow file upon failure (by message) - and you can always inspect the messages using 'data provenance' feature, and replay the messages selectively, if the need be.

Re: Executing apache NIFI through a shell script

Explorer

Can you provide an example or any link where we are programming via NIFI API

Highlighted

Re: Executing apache NIFI through a shell script

Hi @INDRANIL ROY,

Take a look at NiFi Api documentation below, you can control the processors and do a lot more:

https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

For examples to start stop and change configuration, check out my answer to below question in HCC

https://community.hortonworks.com/questions/41532/nifi-job-control-and-date-parameter.html#answer-41...

Thanks,

Jobin George

Don't have an account?
Coming from Hortonworks? Activate your account here