@bowen, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future. ... View more

@Chetankc  From a NiFi perspective there is not much guidance that can be given with such little information. What does "10 Billion Load" mean?  Is it the number if unique files being ingested to NiFi?  What is size average?  What is rate of ingest?   What is "15,000 process"?  Is this the number of NiFi processors added to the NiFi canvas?  What types of processors are being used?  Does your dataflow(s) do a lot of content modification? Have you done testing on throughput performance and done any performance tuning?  15,000 processors is a lot of execution scheduling against your CPU cores. In your load testing what was you CPU load average?  What was your memory impact?  You also have custom NiFi components.  Are you referring to these custom components as using many threads or the totality of the 15,000 components using a lot of threads? What does a lot of threads mean here? Are any of these long running threads or are they all millisecond thread executions? What kind of performance and throughput are you achieving now?   and onn what type of setup (how many nodes in your NiFi cluster, number of CPU cores, JVM Heap settings, type of disk, etc) currently?  Thank you, Matt ... View more

@hidden  Welcome to the world of Apache NiFi.  The first recommendation I'd make is to download the latest version of Apache NiFi 1.x branch.  The 1.12 branch is more then 5 years old now and there have been so many improvements, bug fixes, and security updates since its release. The new Apache NiFi 2.x branch has also been released recently.  Since you are new to NiFi you may also consider utilizing the 2.x version instead to avoid hassle down the road of migrating to this new major release branch.  The 1.x branch will cease to release new versions soon. When sharing exceptions for help is is best to make sure you have also inspected the NiFi-Registry logs produced in the log directory you have configured in the logback.xml file.  They may provide more detailed stack traces and/or logging to help fully understand the issue you encountered.   Thank you, Matt ... View more

Hi @MrBeasr  Please check the status of job via Oozie Database: # select count(*) from WF_JOBS where id like '%0043858-220322202819429-oozie-oozi-W%';   If require you may set the status of the job in Database itself: # update wf_jobs set status='FAILED' where id like '%0043858-220322202819429-oozie-oozi-W%';  ... View more

Congratulations ... View more

@Choolake, Did the response assist in resolving your query? If it did, kindly mark the relevant reply as the solution, as it will aid others in locating the answer more easily in the future.  ... View more

I suspect one more NiFi service is running on the same host with the same port, It can be checked easily by running "ps -ef | grep nifi"   ... View more