question Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group in Support Questions

Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

s198 — Fri, 05 Apr 2024 18:48:29 GMT

My data flow starts from a single FlowFile produced by a Sqoop job, which then expands into multiple FlowFiles after executing the GetHDFSFileInfo Processor (based on the number of HDFS files). To capture all failure scenarios, I have created a Child processor group with the following processors

Input Port --> GetHDFSFileInfo --> RouteOnAttribute --> UpdateAttribute --> FetchHDFS --> PutSFTP --> ModifyBytes --> Output Port

Main Processor Group
--------------------
RouteOnAttribute --> Above mentioned Child Processor Group --> MergeContent --> downstream processors

The child processor group is configured with "FlowFile Concurrency = Single FlowFile Per Node" and "Outbound Policy = Batch Output" to ensure that all fetched FlowFiles are successfully processed (written to the SFTP server).

My GetHDFSFileInfo processor returns 20 HDFS files, and each execution successfully transfers 18 to 19 files to my SFTP server. However, during each execution, one or two file transfers fail in the PutSFTP Processor with the error message 'Failed to rename dot-file.' An error screenshot is attached below

I am facing this issue only when the child processor is configured with "Outbound Policy = Batch Output".
If we try without child processor group, then also it is working.

Could you please help to fix the issue with putSFTP processor.

This is in continuation with the solution provided in the thread https://community.cloudera.com/t5/Support-Questions/How-to-convert-merge-Many-flow-files-to-single-flow-file-in/m-p/385990#M245919

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

s198 — Tue, 09 Apr 2024 11:49:44 GMT

1) Initially, I faced the "NiFi PutSFTP failed to rename dot file issue" only when the child processor was configured with "Outbound Policy = Batch Output". It worked without the child processor group.

2) I modified the PutSFTP failure retry attempt to 3, and it fixed the issue.

3) Later, I introduced a RouteOnAttribute after the FetchHDFS processor for some internal logic implementation, and the PutSFTP error started again.

4) This time, I updated the "Run Schedule" of the PutSFTP processor from 0Sec to 3 Sec, and it again fixed the issue.

5) I have a requirement to transfer stats of each file (with file name, row count, file size) etc. So, I introduced one more PutSFTP processor, and the issue popped up again.

6) Finally, I made the following changes to both of my PutSFTP processors:

a) Added PutSFTP failure retry attempt to 3.

b) Modified the "Run Schedule" of the first PutSFTP Processor to "7 Sec".

c) Modified the "Run Schedule" of the second PutSFTP Processor to "10 Sec".

Now it is working fine. Are we getting this issue because of 20 flowfiles processing at a time ? Could you please suggest if this is the right way to fix the "NiFi PutSFTP failed to rename dot file issue"?

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

VidyaSargur — Fri, 12 Apr 2024 04:44:38 GMT

@TimothySpann @steven-matison @SAMSAL , would you be able to help @s198 please?

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

TimothySpann — Sat, 13 Apr 2024 15:55:08 GMT

It looks like a timing or multiple components accessing the same sftp server. sFTP is very slow and nonthreaded and timing of accessing them can be tricky.

I recommend setting only 1 PutSFTP instance (dont run on cluster and don't have more than one post at once).

We need to see logs and know operating system, java version, NiFi version and some error details. File system type , sftp server version and the type and size of the HDFS files.

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

s198 — Mon, 15 Apr 2024 11:40:21 GMT

Hi @TimothySpann

Please find below the requested info.

Operating system:

Linux

Java version :

Nifi Server:

openjdk version "11.0.22" 2024-01-16 LTS

OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS)

OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS, mixed mode, sharing)

SFTP Server:

openjdk version "1.8.0_402"

OpenJDK Runtime Environment (build 1.8.0_402-b06)

OpenJDK 64-Bit Server VM (build 25.402-b06, mixed mode)

NiFi version:

Cloudera Flow Management (CFM) 2.1.5.1027

1.18.0.2.1.5.1027-2 built 02/09/2023 22:16:12 CST

Tagged nifi-1.18.0-RC4

File system type:

HDFS

Sftp server version :

OpenSSH_7.4p1, OpenSSL 1.0.2k-fips

Type and size of the HDFS files:

I am trying to transfer files ranging in size from 300KB to 800KB. Typically, my HDFS path contains a total of 20 files. In production, the file sizes may vary from 300MB to 600MB, and the total file count would still be 20

And I am running on a Nifi Clustor

I am intermittently facing this failure 'Failed to rename dot-file.' for one or two files while preforming PutSFTP

The first PutSFTP processor is used to transfer the actual file and the second one is used to transfer the stats file corresponding to that file like file name, size, row count etc.

I can limit the second PutSFTP processor to transfer it once with all the 20 files details. ie, Transfer one stats file with the details of all 20 files. Can we store like this info in a variable line by line and then send at the end ?
FileName~RowCount~FileSize
file1~100~1250
file2~200~3000
The above will also satisfy my requirement instead of multiple stats files for the second PutSFTP processor.

Could you pleae some inputs on this issue.
Thank you

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

TimothySpann — Mon, 15 Apr 2024 17:55:54 GMT

How much RAM on the machine and how much is dedicated in the NiFi configuration (or Cloudera manager)? Should be at least 8GB of RAM.

Either the network is slow or you don't have enough RAM.

You can build files from the stats very easily. That is a good strategy.

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

s198 — Thu, 18 Apr 2024 09:27:15 GMT

Hi @TimothySpann

If we introduce any "Run Schedule" delay for the putSFTP processor will it help to fix this issue ? I mean to change the "Run Schedule" from 0 seconds to 30 seconds or something?

There is no network delay and regarding the RAM size, we are yet to hear back from the platform team.

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

TimothySpann — Thu, 18 Apr 2024 11:14:50 GMT

delay could help

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

MattWho — Thu, 18 Apr 2024 13:10:53 GMT

@s198

Question for you:
- If you leave the putSFTP processor stopped, run your dataflow so all FlowFiles queue in front of the putSFTP processor and then start the putSFTP processor, does the issue still happen?
- Does issue only happen when the flow is an all started/running state?

Answers to above can help in determining if changing the run schedule will help here.

Run Schedule details:
- The run schedule works in conjunction with Timer Driven scheduling strategy. This schedule setting controls how often a component will get scheduled to execute (different from when it actually executes. Execution depends on available threads in the NiFi Timer Driven thread pool shared by all components). By default this is set to 0 secs which means that NiFi should schedule his processor as often as possible (Basically schedule it again as soon as an available concurrent task (concurrent tasks default is 1) is available to it. To avoid CPU saturation here, NiFi builds in a yield duration if upon scheduling of a processor there is no work to be done (inbound connection(s) are empty).

Depending on load on yoru system and dataflow, speed of network, this could happen very quick meaning it scheduled, sees only one FlowFile in the inbound connection at time of schedule and processes only that one FlowFile instead of a batch. It then closes that thread and starts a new one for next FlowFile instead of processing multiple FlowFiles in one SFTP connection. By changing run schedule you are allowing more time between scheduling for FlowFiles to queue on the inbound connection so they get batch processed in a single SFTP connection.

Run Duration details:
Another option on processors is the run duration setting. What this adjustment does is upon scheduling of a processor the execution will not end until the configured run duration has elapsed. So lets say at time of scheduling (run schedule) there is one FlowFile in inbound connection queue (remember we are dealing with micro seconds here, so not something you can visualize yourself via the UI). That execution thread will execute against that FlowFile, but rather then close out the session immediately committing the FlowFile to an outbound relationship, it will check inbound connection for another FlowFile and process it in same session. It will continue to do this until the run duration is satisfied at which time all processed FlowFiles during that execution are committed to downstream relationship(s). So Run Duration might be another setting you try to see if it helps with your issue. If you try run duration, i'd set run schedule to default.

You may also want to look at your SFTP server logs to see what is happening when the file rename attempts are failing.

Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

s198 — Thu, 18 Apr 2024 19:01:57 GMT

Hi @MattWho

Thank you for the great suggestions and super helpful information.

Here are the results of what I tried:

I set the Run Schedule to 0 seconds and stopped the PutSFTP processor. After all 20 flowfiles were queued up, I started it again.
Result: Out of 20 flow files, 1 failed.
I set the Run Schedule to 0 seconds and let the flow run with all processors started (Here also, all 20 flow files came almost same time)
Result: Out of 20 flow files, 2 failed.
I updated the Run Schedule of the PutSFTP processor from 0 seconds to 30 seconds.
Result: No failures, all 20 flow files passed.
I updated the "Run Duration" to 500ms.
Result: No failures, all 20 flow files passed.

Could you please suggest the best approach to address this scenario? Option 3 or 4

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

MattWho — Thu, 18 Apr 2024 20:30:23 GMT

@s198
I think step one would be looking more into the failures. Are the failures always with rename of dot file? Put SFTP is writing to a dot file (hidden file) and then upon write completion moves file from .xyz to xyz. You also never shared your complete putSFTP processor configuration.

1. Did you inspect the SFTP server log for any logging related to the failures you encountered?
2. What is being done with the files once placed on the SFTP server? Is there some other process consuming them from there?
3. Any chance that other process is consuming the dot files (hidden files) before NiFi has a chance to rename them?
4. Any of the FlowFiles queued have the same "filename" attribute as another FlowFile or a file already present on the target SFTP server? (this is a common issue where the file of same name still exists on the target when the other is written as dot file and then rename fails. Then on retry some process consumed the duplicate and the new is then successful on rename).

As far as option 3 and 4 go, both introduce some latency in your dataflow.
with (3) the processor only get scheduled once every 30 seconds. So FlowFiles will queue up every 30 seconds. The putSFTP processor has a batch setting for how many FlowFiles will get processed in that execution. If more FlowFiles are queued then that batch setting, he extras will sit until next time processor is scheduled.

My concern is that latency introduced my options 3 and 4 may simply be masking the actual issue needing to be addressed.

With (4) the processor gets scheduled as fast as possible, but when it executes the thread remains active for 500ms working on as many FlowFiles as possible in the single execution. Then at 500ms it close out that thread and the processor (assuming run schedule of 0) would immediate schedule the processor again.

As far as which is better, it is about getting best performance throughput with least amount of latency. Data volumes, sizes, etc come it play here. I typically favor option 4 myself. But if option 3 still works for you but with a much lower runs schedule (30 secs is a lot of latency for a continues flow)

Thank you,
Matt

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

s198 — Mon, 22 Apr 2024 18:51:45 GMT

Hi @MattWho

Thanks again for your suggestions

The failures always occur during the renaming of dot files.

There is no process consuming the file once it is placed on the SFTP server, and there is no chance that another process is consuming the dot files. None of the queued FlowFiles have the same 'filename' attribute as another FlowFile or a file already present on the target SFTP server.

Unfortunately, we don't have permission to view the SFTP logs on our Linux server. I will connect with the admin team to obtain sample logs.

PFB my putSFTP Processor configuration

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

MattWho — Mon, 22 Apr 2024 19:11:44 GMT

@s198

- Do you have the full stack trace from the nifi-app.log when the rename fails?
- Is it always the same exact stack trace?
- Have tried putting this processor class in DEBUG via the NiFi logback.xml to see what additional logging it may produce when the exception occurs?

Thanks,
Matt

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

s198 — Tue, 23 Apr 2024 06:56:51 GMT

Hi @MattWho

Apologies!

I realized that the failed FlowFiles have the same 'filename' attribute as another FlowFile that has already been transferred to the target SFTP server.

There are 20 files available in my HDFS path, and once the file is fetched, I am updating the 'filename' attribute using the below expression. However, one or two files are getting the same 'filename,' which is causing the error 'failed to rename dot-file':

${ExtractName}_${now():format('yyyyMMddHHmmssSSS')}.txt

Thank you for providing inputs to identify the root cause of this issue. Now, could you please suggest the best approach to avoid this scenario, as even with milliseconds, it is getting the same filename?

As you suggested earlier, can we go with "Run Duration" setting for putSFTP processor

Thank you

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

MattWho — Tue, 23 Apr 2024 12:56:10 GMT

@s198

The two most common scenarios for this type of failure are:
1. File already exists with same name when trying to rename. Typically resolved by using an update attribute when a failure exists to modify the filename. Perhaps use the nextInt NiFi expression Language function to add an incremental number to filename or in your case modify the time by adding a few milliseconds to it.
2. Some process is consuming the dot (.) filename before the putSFTP processor has renamed it. This requires modifying the downstream process to ignore dot files.

While it is great that run duration and run schedule increases appear to resolve this issue, I think you are dealing will a millisecond raise condition and these two options will not always guarantee success here. Best option is to programmatically deal with the failures with a filename attribute modification or change who you are uniquely naming your files if possible.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

TimothySpann — Tue, 23 Apr 2024 14:35:11 GMT

#4 - no failures. maybe your ftp server can't handle more than that

Re: Nifi PutSFTP failed to rename dot file when "Outbound Policy = Batch Output" for a child processor group

s198 — Tue, 23 Apr 2024 14:56:08 GMT

Thank you, @MattWho for providing timely responses and quick solutions to the queries. You are really helping the community grow. Hats off to you. Appreciate it