Member since
03-01-2017
30
Posts
2
Kudos Received
0
Solutions
10-08-2022
09:16 AM
Hi, I am using NiFi 1.14.0 version. I installed it within podman having openshift. The problem which I am facing is after sometime NiFi stuck and unable to serve any request. Moreover new flowfiles are not generating and threads are stuck on Running processor. As per initial debug when checked on java melody, found 'Provenance Repository maintenance' and 'Lucene Index' thread are in Blocked state. Please assist on this. Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
05-15-2017
12:31 PM
It means there is no way to get success/failure status of spark job if node executing job goes down.
... View more
05-15-2017
11:32 AM
I setup three node nifi cluster. I created custom SparkJobExecutor processor. My workflow is something like this, It take input parameters from one processor and pass this to SparkJobExecutor processor and wait for either success/failure returned by SparkJobExecutor processor. Based on success/failure returned by SparkJobExecutor , my further flow will be decided. As there is a cluster, if one node triggers Spark job, and wait for its response(it will take something 3-4 hrs.) and in between if that node goes down, Can we get final status(fail/success) to other nodes in cluster??
... View more
Labels:
- Labels:
-
Apache NiFi
05-11-2017
03:26 PM
@Matt Clarke If one flowfile is acquire by any processor, then all the processing is done by that processor only or there is an option to distribute it on other nodes, if working in an cluster.
... View more
05-11-2017
01:42 PM
Hi Matt, Can you briefly explain what this line exactly mean "The distributed files are then fed to a FetchSFTP processor that will retrieve the content form the SFTP server and insert it in to the FlowFile at that time"
... View more
05-11-2017
10:52 AM
flow.pngThanks Matt, All above steps are helpful for me to setup cluster. I attach my workflow. Workflow is configured so that GetSFTP is executed on primary node and after that there is Remote process group. By GetSFTP all file contents are put on primary node local content repository. So when flow reaches to remote process group,it will distribute flowfiles to nodes connected in cluster. As each node read contents from their local content repository, so in case of RPG , Is contents are again written over nodes local contents repository? If so, then suppose a case where GetSFTP gets multiple file having 2tb total size and write all contents in primary node content repo. After that RPG distribute flowfiles among different nodes, and again it will write contents in their respective repo. So this will be overhead of space as well as time.
... View more
05-10-2017
08:37 AM
Hi Matt, As in cluster, each node having its own flow.xml.gz and repo, and suppose if one node having 100 flowfiles to process. Can this node transfer flowfiles to another node in cluster? And if this node is not transfering flowfile to other node, then what is the use of clustering?
... View more