Member since
11-25-2021
17
Posts
1
Kudos Received
0
Solutions
04-16-2023
11:17 PM
1 Kudo
Due to space constraints, it is not bundled by default. You can however use it. Follow the below link https://jd-bots.com/2021/09/10/fixed-inferavroschema-not-found-in-latest-version-of-nifi-1-10-0-and-above/
... View more
03-27-2023
05:38 AM
I have a requirement where I need to execute a SQL query to a DB and transform the returned avro file to XML. First I need to replace a few of the records in the file. So I used a split record processor with Avro reader and XML writer to split the records Two replace text processors to replace the required texts in each record Then merge the files into a single file Once the file is merged, I need to replace the XML declaration and root recrod of XML with custom values. The resultant file is about 2GB in size. Can I use sed 1,2d command in execute stream command processor to remove the files and replace text processor to prepend the required text. If I use this method, will the large file occupy system memory and affect performance of other pipelines? Flow is attached
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
02-20-2023
02:54 AM
I have a setup where there is 1 S3 bucket for each day and 4k files per bucket. The requirement I have is 1) Compress all the files 2) archive the files 3) push the files to different bucket 4) delete the source files after the success of the above three operations. Please suggest a nifi flow for this requirement. What I have tried: I tried listing the files using ListS3, Fetching the files using FetchS3, Compress content and tar them using merge content processor. After merge I could not get the filenames to delete from source bucket.
... View more
Labels:
- Labels:
-
Apache NiFi
11-17-2022
12:59 AM
Hi @hargav I had similar issues as well. If you could share your configuration, I can help out Initially, check what is your listing strategy, have you turned on recursive searching, target system timestamp precision etc., incase your issue is related to ListSFTP
... View more
11-08-2022
10:48 PM
Thanks @MattWho This has solved the problem.
... View more
11-07-2022
04:47 AM
Hi @ckumar Yes I have verified. The port is not being used. However nifi throws the error.
... View more
11-07-2022
01:56 AM
I need to monitor the flowfiles that are queued for more than a day in NiFi. Is there a way to monitor that automatically? I have about 1400 processors running in production. It is very tedious to monitor it manually.
... View more
Labels:
- Labels:
-
Apache NiFi
11-07-2022
01:20 AM
I'm trying to run listenHTTP processor for listening to a webhook notification. I'm relatively new to this processor. When I initialize, I observe java.net.bind exception. Address already in use. nifi url: http://localhost:8080/nifi/ (it has a public IP but we use NATing) Existing configuration Error Observed
... View more
Labels:
- Labels:
-
Apache NiFi
11-07-2022
01:03 AM
Hi @MattWho Please find my responses - Does your target SFTP path have multiple sub-directories which are being searched? Is Search Recursively set to "true"? --> Search recursively is set to true - Are there symlink directories in use? - No - Have you looked the state recorded timestamp for your SFTP server directories? Are your missed files having older timestamps? -- Missing files have older time stamps. - How many files average are being written to the target SFTP between 12am and 1am each day? file count ranges from 10 - 100 and all these files are not being picked. Why not just have your listSFTP processor run all the time? Is this because timestamps are not being updated consistently? Even running the processor all the time with cron schedule is not picking the files.
... View more
10-28-2022
12:39 AM
Problem: I have set up a listSFTP processor to run between 2am to 2pm everyday. But the files are not being picked up by the processor. Existing setup and configuration: The files are generated at the SFTP server at 12 am to 1am every day. ListSFTP Configuration Schedule: listing strategy: tracking timestamps other tracking configuration When I start this processor, it runs as expected for 1 day or 2 and after that, the files are not picked. Is it the tracking time window of 3 hours that is affecting the file being listed?
... View more
Labels:
- Labels:
-
Apache NiFi