Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

[#1]

My data flow starts from a single FlowFile produced by a Sqoop job, which then expands into multiple FlowFiles after executing the GetHDFSFileInfo Processor (based on the number of HDFS files). To capture all failure scenarios, I have created a Child processor group with the following processors

Input Port --> GetHDFSFileInfo --> RouteOnAttribute --> UpdateAttribute --> FetchHDFS --> PutSFTP --> ModifyBytes --> Output Port

Main Processor Group
--------------------
RouteOnAttribute --> Above mentioned Child Processor Group --> MergeContent --> downstream processors

The child processor group is configured with "FlowFile Concurrency = Single FlowFile Per Node" and "Outbound Policy = Batch Output" to ensure that all fetched FlowFiles are successfully processed (written to the SFTP server).

My GetHDFSFileInfo processor returns 20 HDFS files, and each execution successfully transfers 18 to 19 files to my SFTP server. However, during each execution, one or two file transfers fail in the PutSFTP Processor with the error message 'Failed to rename dot-file.' An error screenshot is attached below

I am facing this issue only when the child processor is configured with "Outbound Policy = Batch Output".
If we try without child processor group, then also it is working.

Could you please help to fix the issue with putSFTP processor.

This is in continuation with the solution provided in the thread https://community.cloudera.com/t5/Support-Questions/How-to-convert-merge-Many-flow-files-to-single-f...

[#2]

@s198 
I think step one would be looking more into the failures.  Are the failures always with rename of dot file?  Put SFTP is writing to a dot file (hidden file) and then upon write completion moves file from .xyz to xyz.   You also never shared your complete putSFTP processor configuration. 

1. Did you inspect the SFTP server log for any logging related to the failures you encountered?
2. What is being done with the files once placed on the SFTP server?  Is there some other process consuming them from there?
3. Any chance that other process is consuming the dot files (hidden files) before NiFi has a chance to rename them?
4. Any of the FlowFiles queued have the same "filename" attribute as another FlowFile or a file already present on the target SFTP server?  (this is a common issue where the file of same name still exists on the target when the other is written as dot file and then rename fails.  Then on retry some process consumed the duplicate and the new is then successful on rename).

As far as option 3 and 4 go, both introduce some latency in your dataflow.
with (3) the processor only get scheduled once every 30 seconds.  So FlowFiles will queue up every 30 seconds.  The putSFTP processor has a batch setting for how many FlowFiles will get processed in that execution.  If more FlowFiles are queued then that batch setting, he extras will sit until next time processor is scheduled.

My concern is that latency introduced my options 3 and 4 may simply be masking the actual issue needing to be addressed.

With (4) the processor gets scheduled as fast as possible, but when it executes the thread remains active for 500ms working on as many FlowFiles as possible in the single execution.  Then at 500ms it close out that thread and the processor (assuming run schedule of 0) would immediate schedule the processor again.

As far as which is better, it is about getting best performance throughput with least amount of latency.    Data volumes, sizes, etc come it play here.  I typically favor option 4 myself.  But if option 3 still works for you but with a much lower runs schedule (30 secs is a lot of latency for a continues flow) 

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#3]

Hi @MattWho 

Thanks again for your suggestions

The failures always occur during the renaming of dot files.

There is no process consuming the file once it is placed on the SFTP server, and there is no chance that another process is consuming the dot files. None of the queued FlowFiles have the same 'filename' attribute as another FlowFile or a file already present on the target SFTP server.

Unfortunately, we don't have permission to view the SFTP logs on our Linux server. I will connect with the admin team to obtain sample logs.

PFB my putSFTP Processor configuration

[#4]

@s198 

- Do you have the full stack trace from the nifi-app.log when the rename fails?
- Is it always the same exact stack trace?
- Have tried putting this processor class in DEBUG via the NiFi logback.xml to see what additional logging it may produce when the exception occurs?

Thanks,
Matt

[#5]

Hi @MattWho 

Apologies!

I realized that the failed FlowFiles have the same 'filename' attribute as another FlowFile that has already been transferred to the target SFTP server.

There are 20 files available in my HDFS path, and once the file is fetched, I am updating the 'filename' attribute using the below expression. However, one or two files are getting the same 'filename,' which is causing the error 'failed to rename dot-file':

${ExtractName}_${now():format('yyyyMMddHHmmssSSS')}.txt

Thank you for providing inputs to identify the root cause of this issue. Now, could you please suggest the best approach to avoid this scenario, as even with milliseconds, it is getting the same filename? 

As you suggested earlier, can we go with "Run Duration"  setting for putSFTP processor

Thank you

[#6 · Accepted solution]

@s198 

The two most common scenarios for this type of failure are:
1. File already exists with same name when trying to rename.  Typically resolved by using an update attribute when a failure exists to modify the filename.  Perhaps use the nextInt NiFi expression Language function to add an incremental number to filename or in your case modify the time by adding a few milliseconds to it.
2. Some process is consuming the dot (.) filename before the putSFTP processor has renamed it.  This requires modifying the downstream process to ignore dot files.

While it is great that run duration and run schedule increases appear to resolve this issue, I think you are dealing will a millisecond raise condition and these two options will not always guarantee success here.  Best option is to programmatically deal with the failures with a filename attribute modification or change who you are uniquely naming your files if possible.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#7]

#4 - no failures.   maybe your ftp server can't handle more than that

[#8]

Thank you, @MattWho  for providing timely responses and quick solutions to the queries. You are really helping the community grow. Hats off to you. Appreciate it