Silver ⭐️⭐️⭐️

Solved

Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

16 replies
16074 views
Translate All

My data flow starts from a single FlowFile produced by a Sqoop job, which then expands into multiple FlowFiles after executing the GetHDFSFileInfo Processor (based on the number of HDFS files). To capture all failure scenarios, I have created a Child processor group with the following processors

Input Port --> GetHDFSFileInfo --> RouteOnAttribute --> UpdateAttribute --> FetchHDFS --> PutSFTP --> ModifyBytes --> Output Port

Main Processor Group
--------------------
RouteOnAttribute --> Above mentioned Child Processor Group --> MergeContent --> downstream processors

The child processor group is configured with "FlowFile Concurrency = Single FlowFile Per Node" and "Outbound Policy = Batch Output" to ensure that all fetched FlowFiles are successfully processed (written to the SFTP server).

My GetHDFSFileInfo processor returns 20 HDFS files, and each execution successfully transfers 18 to 19 files to my SFTP server. However, during each execution, one or two file transfers fail in the PutSFTP Processor with the error message 'Failed to rename dot-file.' An error screenshot is attached below

I am facing this issue only when the child processor is configured with "Outbound Policy = Batch Output".
If we try without child processor group, then also it is working.

Could you please help to fix the issue with putSFTP processor.

This is in continuation with the solution provided in the thread https://community.cloudera.com/t5/Support-Questions/How-to-convert-merge-Many-flow-files-to-single-flow-file-in/m-p/385990#M245919

MattWhoAnswer

Titanium ⭐️⭐️⭐️

@s198

The two most common scenarios for this type of failure are:
1. File already exists with same name when trying to rename. Typically resolved by using an update attribute when a failure exists to modify the filename. Perhaps use the nextInt NiFi expression Language function to add an incremental number to filename or in your case modify the time by adding a few milliseconds to it.
2. Some process is consuming the dot (.) filename before the putSFTP processor has renamed it. This requires modifying the downstream process to ignore dot files.

While it is great that run duration and run schedule increases appear to resolve this issue, I think you are dealing will a millisecond raise condition and these two options will not always guarantee success here. Best option is to programmatically deal with the failures with a filename attribute modification or change who you are uniquely naming your files if possible.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

s198Author

Silver ⭐️⭐️⭐️

1) Initially, I faced the "NiFi PutSFTP failed to rename dot file issue" only when the child processor was configured with "Outbound Policy = Batch Output". It worked without the child processor group.

2) I modified the PutSFTP failure retry attempt to 3, and it fixed the issue.

3) Later, I introduced a RouteOnAttribute after the FetchHDFS processor for some internal logic implementation, and the PutSFTP error started again.

4) This time, I updated the "Run Schedule" of the PutSFTP processor from 0Sec to 3 Sec, and it again fixed the issue.

5) I have a requirement to transfer stats of each file (with file name, row count, file size) etc. So, I introduced one more PutSFTP processor, and the issue popped up again.

6) Finally, I made the following changes to both of my PutSFTP processors:

a) Added PutSFTP failure retry attempt to 3.

b) Modified the "Run Schedule" of the first PutSFTP Processor to "7 Sec".

c) Modified the "Run Schedule" of the second PutSFTP Processor to "10 Sec".

Now it is working fine. Are we getting this issue because of 20 flowfiles processing at a time ? Could you please suggest if this is the right way to fix the "NiFi PutSFTP failed to rename dot file issue"?

Translate

TimothySpann

Platinum ⭐️⭐️⭐️

It looks like a timing or multiple components accessing the same sftp server. sFTP is very slow and nonthreaded and timing of accessing them can be tricky.

I recommend setting only 1 PutSFTP instance (dont run on cluster and don't have more than one post at once).

We need to see logs and know operating system, java version, NiFi version and some error details. File system type , sftp server version and the type and size of the HDFS files.

Translate

VidyaSargur

Community Manager

@TimothySpann @steven-matison @SAMSAL , would you be able to help @s198 please?

Translate

Regards, Vidya Sargur, Community Manager Was your question answered? Make sure to mark the answer as the accepted solution. If you find a reply useful, say thanks by clicking on the thumbs up button.

s198Author

Silver ⭐️⭐️⭐️

Hi @TimothySpann

Please find below the requested info.

Operating system:

Linux

Java version :

Nifi Server:

openjdk version "11.0.22" 2024-01-16 LTS

OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS)

OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS, mixed mode, sharing)

SFTP Server:

openjdk version "1.8.0_402"

OpenJDK Runtime Environment (build 1.8.0_402-b06)

OpenJDK 64-Bit Server VM (build 25.402-b06, mixed mode)

NiFi version:

Cloudera Flow Management (CFM) 2.1.5.1027

1.18.0.2.1.5.1027-2 built 02/09/2023 22:16:12 CST

Tagged nifi-1.18.0-RC4

Powered by Apache NiFi 1.18.0

File system type:

HDFS

Sftp server version :

OpenSSH_7.4p1, OpenSSL 1.0.2k-fips

Type and size of the HDFS files:

I am trying to transfer files ranging in size from 300KB to 800KB. Typically, my HDFS path contains a total of 20 files. In production, the file sizes may vary from 300MB to 600MB, and the total file count would still be 20

And I am running on a Nifi Clustor

I am intermittently facing this failure 'Failed to rename dot-file.' for one or two files while preforming PutSFTP

The first PutSFTP processor is used to transfer the actual file and the second one is used to transfer the stats file corresponding to that file like file name, size, row count etc.

I can limit the second PutSFTP processor to transfer it once with all the 20 files details. ie, Transfer one stats file with the details of all 20 files. Can we store like this info in a variable line by line and then send at the end ?
FileName~RowCount~FileSize
file1~100~1250
file2~200~3000
The above will also satisfy my requirement instead of multiple stats files for the second PutSFTP processor.

Could you pleae some inputs on this issue.
Thank you

Translate

TimothySpann

Platinum ⭐️⭐️⭐️

How much RAM on the machine and how much is dedicated in the NiFi configuration (or Cloudera manager)? Should be at least 8GB of RAM.

Either the network is slow or you don't have enough RAM.

You can build files from the stats very easily. That is a good strategy.

Translate

s198Author

Silver ⭐️⭐️⭐️

Hi @TimothySpann

If we introduce any "Run Schedule" delay for the putSFTP processor will it help to fix this issue ? I mean to change the "Run Schedule" from 0 seconds to 30 seconds or something?

There is no network delay and regarding the RAM size, we are yet to hear back from the platform team.

Translate

TimothySpann

Platinum ⭐️⭐️⭐️

delay could help

Translate

s198Author

Silver ⭐️⭐️⭐️

Hi @MattWho

Thank you for the great suggestions and super helpful information.

Here are the results of what I tried:

I set the Run Schedule to 0 seconds and stopped the PutSFTP processor. After all 20 flowfiles were queued up, I started it again.
Result: Out of 20 flow files, 1 failed.
I set the Run Schedule to 0 seconds and let the flow run with all processors started (Here also, all 20 flow files came almost same time)
Result: Out of 20 flow files, 2 failed.
I updated the Run Schedule of the PutSFTP processor from 0 seconds to 30 seconds.
Result: No failures, all 20 flow files passed.
I updated the "Run Duration" to 500ms.
Result: No failures, all 20 flow files passed.

Could you please suggest the best approach to address this scenario? Option 3 or 4

Translate

MattWho

Titanium ⭐️⭐️⭐️

@s198
I think step one would be looking more into the failures. Are the failures always with rename of dot file? Put SFTP is writing to a dot file (hidden file) and then upon write completion moves file from .xyz to xyz. You also never shared your complete putSFTP processor configuration.

1. Did you inspect the SFTP server log for any logging related to the failures you encountered?
2. What is being done with the files once placed on the SFTP server? Is there some other process consuming them from there?
3. Any chance that other process is consuming the dot files (hidden files) before NiFi has a chance to rename them?
4. Any of the FlowFiles queued have the same "filename" attribute as another FlowFile or a file already present on the target SFTP server? (this is a common issue where the file of same name still exists on the target when the other is written as dot file and then rename fails. Then on retry some process consumed the duplicate and the new is then successful on rename).

As far as option 3 and 4 go, both introduce some latency in your dataflow.
with (3) the processor only get scheduled once every 30 seconds. So FlowFiles will queue up every 30 seconds. The putSFTP processor has a batch setting for how many FlowFiles will get processed in that execution. If more FlowFiles are queued then that batch setting, he extras will sit until next time processor is scheduled.

My concern is that latency introduced my options 3 and 4 may simply be masking the actual issue needing to be addressed.

With (4) the processor gets scheduled as fast as possible, but when it executes the thread remains active for 500ms working on as many FlowFiles as possible in the single execution. Then at 500ms it close out that thread and the processor (assuming run schedule of 0) would immediate schedule the processor again.

As far as which is better, it is about getting best performance throughput with least amount of latency. Data volumes, sizes, etc come it play here. I typically favor option 4 myself. But if option 3 still works for you but with a much lower runs schedule (30 secs is a lot of latency for a continues flow)

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

s198Author

Silver ⭐️⭐️⭐️

Hi @MattWho

Thanks again for your suggestions

The failures always occur during the renaming of dot files.

There is no process consuming the file once it is placed on the SFTP server, and there is no chance that another process is consuming the dot files. None of the queued FlowFiles have the same 'filename' attribute as another FlowFile or a file already present on the target SFTP server.

Unfortunately, we don't have permission to view the SFTP logs on our Linux server. I will connect with the admin team to obtain sample logs.

PFB my putSFTP Processor configuration

Translate

MattWho

Titanium ⭐️⭐️⭐️

@s198

- Do you have the full stack trace from the nifi-app.log when the rename fails?
- Is it always the same exact stack trace?
- Have tried putting this processor class in DEBUG via the NiFi logback.xml to see what additional logging it may produce when the exception occurs?

Thanks,
Matt

Translate

s198Author

Silver ⭐️⭐️⭐️

Hi @MattWho

Apologies!

I realized that the failed FlowFiles have the same 'filename' attribute as another FlowFile that has already been transferred to the target SFTP server.

There are 20 files available in my HDFS path, and once the file is fetched, I am updating the 'filename' attribute using the below expression. However, one or two files are getting the same 'filename,' which is causing the error 'failed to rename dot-file':

${ExtractName}_${now():format('yyyyMMddHHmmssSSS')}.txt

Thank you for providing inputs to identify the root cause of this issue. Now, could you please suggest the best approach to avoid this scenario, as even with milliseconds, it is getting the same filename?

As you suggested earlier, can we go with "Run Duration" setting for putSFTP processor

Thank you

Translate

MattWhoAnswer

Titanium ⭐️⭐️⭐️

@s198

The two most common scenarios for this type of failure are:
1. File already exists with same name when trying to rename. Typically resolved by using an update attribute when a failure exists to modify the filename. Perhaps use the nextInt NiFi expression Language function to add an incremental number to filename or in your case modify the time by adding a few milliseconds to it.
2. Some process is consuming the dot (.) filename before the putSFTP processor has renamed it. This requires modifying the downstream process to ignore dot files.

While it is great that run duration and run schedule increases appear to resolve this issue, I think you are dealing will a millisecond raise condition and these two options will not always guarantee success here. Best option is to programmatically deal with the failures with a filename attribute modification or change who you are uniquely naming your files if possible.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate

TimothySpann

Platinum ⭐️⭐️⭐️

#4 - no failures. maybe your ftp server can't handle more than that

Translate

s198Author

Silver ⭐️⭐️⭐️

Thank you, @MattWho for providing timely responses and quick solutions to the queries. You are really helping the community grow. Hats off to you. Appreciate it

Translate

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.