Thank you, @MattWho  for providing timely responses and quick solutions to the queries. You are really helping the community grow. Hats off to you. Appreciate it ... View more

Hi @MattWho  Apologies! I realized that the failed FlowFiles have the same 'filename' attribute as another FlowFile that has already been transferred to the target SFTP server. There are 20 files available in my HDFS path, and once the file is fetched, I am updating the 'filename' attribute using the below expression. However, one or two files are getting the same 'filename,' which is causing the error 'failed to rename dot-file': ${ExtractName}_${now():format('yyyyMMddHHmmssSSS')}.txt Thank you for providing inputs to identify the root cause of this issue. Now, could you please suggest the best approach to avoid this scenario, as even with milliseconds, it is getting the same filename?  As you suggested earlier, can we go with "Run Duration"  setting for putSFTP processor Thank you ... View more

Hi @MattWho  Thanks again for your suggestions The failures always occur during the renaming of dot files. There is no process consuming the file once it is placed on the SFTP server, and there is no chance that another process is consuming the dot files. None of the queued FlowFiles have the same 'filename' attribute as another FlowFile or a file already present on the target SFTP server. Unfortunately, we don't have permission to view the SFTP logs on our Linux server. I will connect with the admin team to obtain sample logs. PFB my putSFTP Processor configuration     ... View more

Hi @MattWho  Thank you for the great suggestions and super helpful information. Here are the results of what I tried: I set the Run Schedule to 0 seconds and stopped the PutSFTP processor. After all 20 flowfiles were queued up, I started it again. Result: Out of 20 flow files, 1 failed.  I set the Run Schedule to 0 seconds and let the flow run with all processors started (Here also, all 20 flow files came almost same time) Result: Out of 20 flow files, 2 failed. I updated the Run Schedule of the PutSFTP processor from 0 seconds to 30 seconds. Result: No failures, all 20 flow files passed. I updated the "Run Duration" to 500ms. Result: No failures, all 20 flow files passed. Could you please suggest the best approach to address this scenario? Option 3 or 4 ... View more

Hi @TimothySpann  If we introduce any "Run Schedule" delay for the putSFTP processor will it help to fix this issue ? I mean to change the  "Run Schedule" from 0 seconds to 30 seconds or something? There is no network delay and regarding the RAM size, we are yet to hear back from the platform team. ... View more

Hi @TimothySpann    Please find below the requested info.   Operating system:                  Linux Java version :              Nifi Server:                       openjdk version "11.0.22" 2024-01-16 LTS                       OpenJDK Runtime Environment (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS)                      OpenJDK 64-Bit Server VM (Red_Hat-11.0.22.0.7-1) (build 11.0.22+7-LTS, mixed mode, sharing)               SFTP Server:                      openjdk version "1.8.0_402"                         OpenJDK Runtime Environment (build 1.8.0_402-b06)                      OpenJDK 64-Bit Server VM (build 25.402-b06, mixed mode)   NiFi version:                          Cloudera Flow Management (CFM) 2.1.5.1027                        1.18.0.2.1.5.1027-2 built 02/09/2023 22:16:12 CST                       Tagged nifi-1.18.0-RC4                       Powered by Apache NiFi 1.18.0 File system type:                      HDFS    Sftp server version :                   OpenSSH_7.4p1, OpenSSL 1.0.2k-fips   Type and size of the HDFS files:                      I am trying to transfer files ranging in size from 300KB to 800KB. Typically, my HDFS path contains a total of 20 files. In production, the file sizes may vary from 300MB to 600MB, and the total file count would still be 20 And I am running on a Nifi Clustor    I am intermittently facing this failure  'Failed to rename dot-file.' for one or two files while preforming PutSFTP    The first PutSFTP processor is used to transfer the actual file and the second one is used to transfer the stats file corresponding to that file like file name, size, row count etc.  I can limit the second  PutSFTP  processor to transfer it once with all the 20 files details. ie, Transfer one stats file with the details of all 20 files. Can we store like this info in a variable line by line and then send at the end ?  FileName~RowCount~FileSize file1~100~1250 file2~200~3000 The above will also satisfy my requirement instead of multiple stats files for the second PutSFTP processor. Could you pleae some inputs on this issue.  Thank you ... View more

1) Initially, I faced the "NiFi PutSFTP failed to rename dot file issue" only when the child processor was configured with "Outbound Policy = Batch Output". It worked without the child processor group. 2) I modified the PutSFTP failure retry attempt to 3, and it fixed the issue. 3) Later, I introduced a RouteOnAttribute after the FetchHDFS processor for some internal logic implementation, and the PutSFTP error started again.  4) This time, I updated the "Run Schedule" of the PutSFTP processor from 0Sec to 3 Sec, and it again fixed the issue.  5) I have a requirement to transfer stats of each file (with file name, row count, file size) etc. So, I introduced one more PutSFTP processor, and the issue popped up again.  6) Finally, I made the following changes to both of my PutSFTP processors:        a) Added PutSFTP failure retry attempt to 3.        b) Modified the "Run Schedule" of the first PutSFTP Processor to "7 Sec".        c) Modified the "Run Schedule" of the second PutSFTP Processor to "10 Sec".   Now it is working fine. Are we getting this issue because of 20 flowfiles processing at a time ? Could you please suggest if this is the right way to fix the "NiFi PutSFTP failed to rename dot file issue"? ... View more

Hi @MattWho    1) Initially, I faced the "NiFi PutSFTP failed to rename dot file issue" only when the child processor was configured with "Outbound Policy = Batch Output". It worked without the child processor group. 2) I modified the PutSFTP failure retry attempt to 3, and it fixed the issue. 3) Later, I introduced a RouteOnAttribute after the FetchHDFS processor for some internal logic implementation, and the PutSFTP error started again.  4) This time, I updated the "Run Schedule" of the PutSFTP processor from 0Sec to 3 Sec, and it again fixed the issue.  5) I have a requirement to transfer stats of each file (with file name, row count, file size) etc. So, I introduced one more PutSFTP processor, and the issue popped up again.  6) Finally, I made the following changes to both of my PutSFTP processors:        a) Added PutSFTP failure retry attempt to 3.        b) Modified the "Run Schedule" of the first PutSFTP Processor to "7 Sec".        c) Modified the "Run Schedule" of the second PutSFTP Processor to "10 Sec".   Now it is working fine. Are we getting this issue because of 20 flowfiles processing at a time ?  Could you please suggest if this is the right way to fix the "NiFi PutSFTP failed to rename dot file issue"? ... View more

Hi @MattWho  As you suggested I tried with a child processor group as below with "FlowFile Concurrency = Single FlowFile Per Node" and "Outbound Policy = Batch Output" to ensure that all fetched FlowFiles are successfully processed and start the MergeContent Processor. Input Port --> GetHDFSFileInfo --> RouteOnAttribute --> UpdateAttribute --> FetchHDFS --> PutSFTP --> ModifyBytes --> Output Port My GetHDFSFileInfo processor returns 20 HDFS files, and each execution successfully transfers 18 to 19 files to my SFTP server. However, during each execution, one or two file transfers fail in the PutSFTP Processor with the error message 'Failed to rename dot-file.' An error screenshot is attached below I am facing this issue only when the child processor is configured with "Outbound Policy = Batch Output". If we try without child processor group, then also it is working. Am I missing some configuration settings here? Could you please help to fix the issue with the PutSFTP processor? ... View more