Member since
04-13-2018
44
Posts
0
Kudos Received
0
Solutions
07-02-2019
01:10 AM
@Bryan Bende can you pls suggest
... View more
07-02-2019
01:07 AM
Clustered NiFi of 10 nodes have been recently upgraded to v1.9.2. After that the flow/provenance/content storage is not getting cleaned and are eventually filling up and getting to 100% utilization. This will stop the cluster functioning if the affected node is primary node. As a workaround, the node has to be restarted and then the utilization(/data partition) will reduce to 10% or so. Note: The cluster is docker based.
... View more
Labels:
- Labels:
-
Apache NiFi
06-20-2019
03:07 AM
Hi, Our nifi cluster run on 10 nodes,with an external zookeeper.We had an issue where the port 2888 was blocked for one of the zookeepers and the cluster was not responding.There are 3 zookeepers and rest 2 were still active at that time but it seemed that they were going in loop and stuck.Is there any way by which we can disconnect the zookeeper when port is blocked so that the cluster continues running?
... View more
Labels:
- Labels:
-
Apache NiFi
01-07-2019
12:05 PM
Hi, Iam using Nifi 1.3 version and this is the flow..getsftp->putsftp->putsftp. But at times after the files are picked by getsftp,say if its 190 bytes then putsftp transfers as 0 bytes. It doesnot happen all the time but at times. Please find below the error: 2019-01-07 02:00:01,272 ERROR [Timer-Driven Process Thread-7] o.a.nifi.processors.standard.PutSFTP PutSFTP[id=XXX] PutSFTP[id=XXX] failed to process session due to org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find content for FlowFile: {}
org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find content for FlowFile
at org.apache.nifi.controller.repository.StandardProcessSession.handleContentNotFound(StandardProcessSession.java:3003)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2157)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2106)
at org.apache.nifi.processors.standard.PutFileTransfer.onTrigger(PutFileTransfer.java:127)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.nifi.controller.repository.ContentNotFoundException: Could not find content for StandardContentClaim [resourceClaim=StandardResourceClaim[id=XXX, container=default, section=490], offset=0, length=190]: Stream contained only 0 bytes but should have contained 190
at org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.ensureAllContentRead(FlowFileAccessInputStream.java:49)
at org.apache.nifi.controller.repository.io.FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:84)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:284)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:636)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:540)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:492)
at org.apache.nifi.processors.standard.util.SFTPTransfer.put(SFTPTransfer.java:543)
at org.apache.nifi.processors.standard.PutFileTransfer$1.process(PutFileTransfer.java:135)
at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2136)
... 14 common frames omitted
2019-01-07 02:00:01,272 WARN [Timer-Driven Process Thread-7] o.a.nifi.processors.standard.PutSFTP PutSFTP[id=015b10b1-853c-1b95-d98f-915e12c396bd] Processor Administrat
ively Yielded for 1 sec due to processing failure
2019-01-07 02:00:01,273 WARN [Timer-Driven Process Thread-7] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding PutSFTP[id=XXX] due to uncaught Exception: org.apache.nifi.processor.exception.MissingFlowFileException: Unable to find content for FlowFile Can you please help me in finding why the files are transfered as 0 bytes?
... View more
Labels:
- Labels:
-
Apache NiFi
12-13-2018
04:53 AM
@Matt Clarke we are using Nifi 1.7 version and the above said configurations are not present.I had tried using the groovy script. def flowFiles = session.get(50)
if(!flowFiles || flowFiles.size() < 50) { session.rollback()
} else {
def flowFile = session.create(flowFiles)
session.provenanceReporter.join(flowFiles, flowFile) session.remove(flowFiles) session.transfer(flowFile, REL_SUCCESS)
} But the files are not getting processed,its stuck in queue.Can you please any solution
... View more
12-12-2018
01:06 PM
@Matt Clarke Can you please help me out
... View more
12-12-2018
10:30 AM
@Matt Clarkecan you please help
... View more
12-12-2018
10:29 AM
Hi, I have a flow where it puts 50 files in destination using putsftp processor.I need to make sure that the 50 files has reached destination.I have tried using mergecontent but since its clustered nifi with 10 nodes,its not possible.Can you please suggest a solution.
... View more
Labels:
- Labels:
-
Apache NiFi
12-08-2018
01:49 AM
@Matt BurgessI had the similar scenario where i needed the count of flowfiles generated but the script is not working.The incoming queue is queued with the flowfiles and its not getting processed.Is it because its a clustered nifi.Can you please help?
... View more
12-01-2018
06:35 AM
@Matt ClarkeHi Matt..i tried the above configuration but the 136 files are not getting merged in one go.Its getting merged in number of 20 and 15.I had set the bin age to 5mins.I wanted a single notification after merging the 136 files.Can you please help
... View more
11-27-2018
06:42 AM
Hi, I have a flow wherein there is expected to have 136 flowfiles and after receiving the same,its to be notified via putemail. The flow will be putfile->mergecontent->putemail.My doubt is how we can exactly ensure136 flowfiles . Is the property max group size dependent on the size of flowfiles,as the size of these files varies day by day.So the only option is to set max bin age? @Shu can you please help?
... View more
Labels:
- Labels:
-
Apache NiFi
10-26-2018
11:57 AM
@Matt BurgessThere is only single task and its not importing the CPython libraries.This job was working fine for 6 months now and all of a sudden it started failing with the error above in the executescript processor.
... View more
10-25-2018
04:33 AM
Hi, I had a running executescript processor in python which worked fine till yesterday.But suddenly its throwing error as AttributeError: type object 'java.lang.Thread' has no attribute 'State' in <script> at line number 1. Is there any issue with the processor?
... View more
Labels:
- Labels:
-
Apache NiFi
08-14-2018
03:57 AM
@Shu As you have said above instead of having two triggers,cant have a single one as 0 0/15 2-3 1/1 * ? * which would run from 2am to 3am
... View more
08-14-2018
02:17 AM
there are 2 asterix before ? in first exp and asterix between ?..symbol is not getting pasted here
... View more
08-14-2018
02:16 AM
@Shu thanks for the quick reply..wanted to know what is the difference between 0 0/15 2-3 * * ? and 0 0/15 2-3 1/1 * ? *
... View more
08-14-2018
01:55 AM
Hi, I wanted to setup a processor which needs to run from 2:00 to 3:00 AM (every 15 mins). can you please let me know if this will work: 0 0/15 2-3 * * ?
... View more
Labels:
- Labels:
-
Apache NiFi
08-08-2018
07:06 AM
@Shu thanks for the detailed explaination.If the filename keeps changing like "YYYY_MM_DD_HH_MI_SS.csv" ,how can we mention the same in fetch file's filename
... View more
08-08-2018
02:30 AM
@Shu thanks for the same,so the flow should be like generate flowfile->update attribute->execute stream->list file->fetch file?.I needed all the processing to happen in one flow and not two separate flows.
... View more
08-08-2018
01:48 AM
There is a flow which would generate flowfiles,set input parameters using update attribute and then process it by execute stream command.This would however place the required files in the location as mentioned in update attribute processor.Is there any way which could fetch the processed files directly from the location so as to move the processed files into another location using another processor.
... View more
Labels:
- Labels:
-
Apache NiFi
04-27-2018
10:01 AM
Hi, I have setup a 3 node Nifi cluster.When i started running only one node,i got the connection refused error. The errors is because during startup ZK will not establish quorum until all three nodes have completely started. As a node goes through its startup process it will being begin trying to establish zk quorum between all other zk nodes. Those other nodes may not be running yet if the other nodes are still starting as well, thus producing a lot of ERROR messages.So as a result nifi-app log is getting accumulated with error messages and increase in size.Is there a way to handle this?
... View more
Labels:
- Labels:
-
Apache NiFi
04-27-2018
06:11 AM
@Matt Clarke Thanks Matt for the information..and helping out...it worked
... View more