Member since
02-13-2020
9
Posts
0
Kudos Received
0
Solutions
02-09-2021
05:48 AM
@Umakanth The GetSFTP processor actually creates a verbose listing of all Files form target SFTP for which it will be getting. It then fetches all those files. Unlike the ListSFTP processor, the getSFTP is an older deprecated processor that does not store state. My guess here is that at times the listing is larger then other times or as you mentioned some occasional latency occurs resulting in enough time between creating that list and actually consuming the files, that the source system has moved the listed file before it is grabbed. In that case moving to the newer ListSFTP and FetchSFTP processors will help in handling that scenario. The listing will list all the files it sees and the FetchSFTP will fetch the content for those that have not yet been moved by the source system. The FetchSFTP will still throw an exception for each file it can not find still and route those to the not.found relationship which you can handle programmatically in your NiFi dataflow(s). Thanks, Matt
... View more
09-14-2020
09:20 AM
2 Kudos
@Umakanth From your shared log lines we can see two things: 1. "LOG 1" shows "StandardFlowFileRecord[uuid=345d9b6d-e9f7-4dd8-ad9a-a9d66fdfd902" and "LOG 2" shows "Successfully sent [StandardFlowFileRecord[uuid=f74eb941-a233-4f9e-86ff-07723940f012". This tells us these "RandomFile1154.txt" are two different FlowFiles. So does not look like RPG sent the same FlowFile twice, but rather sent two FlowFiles with each referencing the same content. I am not sure how you have your LogAttribute processor configured, but you should look for the log output produced by these two uuids to learn more about these two FlowFiles. I suspect from your comments you will only find one of these passed through your LogAttribute processor. 2. We can see from both logs that the above two FlowFiles actually point at the exact same content in the content_repository: "LOG 1" --> claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1073154, length=237],offset=0,name=RandomFile1154.txt,size=237] "LOG 2" --> claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1599937413340-1, container=default, section=1], offset=1109014, length=237],offset=0,name=RandomFile1154.txt,size=237] This typically happens when a FlowFile becomes cloned somewhere in your dataflow. For example: when a relationship from a processor is defined twice. Since you saw that GetFile only ingested file once, that rules out GetFile as the source of this duplication. But had it been GetFile, you would have not seen identical claim information. LogAttribute only has a single "success" relationship, so if you had drawn two connections with "Success" relationship defined in both, you would have seen duplicates of every ingested content. So this seems unlikely as well. Next you have your PutFile processor. This processor has both "success" and "failure" relationships. I suspect the "success" relationship is assigned to the connection going to your Remote Process Group" and the "failure" relationship assigned to a connection that loops back on the PutFile itself(?). Now if you had accidentally drawn the "failure" connection twice (one may be stack on top of the other), anytime a FlowFile failed in the putFile it would have been routed to one failure connection and cloned to other failure connection. Then on time they both processed successfully by putFile and you end up with the original and clone sent to your RPG. Hope this helps, Matt
... View more
03-31-2020
08:31 AM
There was a bug [1] in the TLS handshake negotiation process in Apache NiFi 1.11.0 - 1.11.3. NiFi 1.11.4 was released on March 22, 2020, which fixes this issue. I recommend upgrading to the latest version. [1] https://issues.apache.org/jira/browse/NIFI-7223
... View more
03-04-2020
10:55 AM
@Umakanth The API is exposed out of the box, it is not something you need to enable. Every action you take while performing actions within the UI makes a call to the NiFi rest-api. When learning how to use the rest-api calls, you may find using the developer tools in your browser helpful. Open the developer tools while you are accessing your NiFi UI. Then perform some action and you will see those requests being made by your browser to NiFi. In the below example (Using Chrome browser developer tools), I opened NiFi's summary UI from the global menu in the upper right corner of the UI: You'll notice that several requests were made. I can write click on any one of those request and select "Copy as cURL" to copy the full request to the system clipboard. I can then paste the request in a terminal window and see what the rest-api call returns. You will notice that the curl command that is copied will have numerous additional headers (-H) that are not always necessary depending on the rest-api endpoint being used. Example: curl 'http://<nifi-hostname>:<nifi-port>/nifi-api/flow/process-groups/root/status?recursive=true' Of course you will need to parse the rest-api returns yourself to extract in many cases the specific details/stats you want to monitor. Hope this helps, Matt
... View more
02-13-2020
05:52 AM
Thank you for sharing your thoughts around this integration, it really really helps. I will try to do a POC based out of your suggestion
... View more