About Jagapriyan

Jagapriyan · ‎03-27-2023

I have a requirement where I need to execute a SQL query to a DB and transform the returned avro file to XML. First I need to replace a few of the records in the file. So I used a split record processor with Avro reader and XML writer to split the records Two replace text processors to replace the required texts in each record Then merge the files into a single file Once the file is merged, I need to replace the XML declaration and root recrod of XML with custom values. The resultant file is about 2GB in size. Can I use sed 1,2d command in execute stream command processor to remove the files and replace text processor to prepend the required text. If I use this method, will the large file occupy system memory and affect performance of other pipelines? Flow is attached

Jagapriyan · ‎02-20-2023

I have a setup where there is 1 S3 bucket for each day and 4k files per bucket. The requirement I have is 1) Compress all the files 2) archive the files 3) push the files to different bucket 4) delete the source files after the success of the above three operations. Please suggest a nifi flow for this requirement. What I have tried: I tried listing the files using ListS3, Fetching the files using FetchS3, Compress content and tar them using merge content processor. After merge I could not get the filenames to delete from source bucket.

Jagapriyan · ‎11-17-2022

Hi @hargav I had similar issues as well. If you could share your configuration, I can help out Initially, check what is your listing strategy, have you turned on recursive searching, target system timestamp precision etc., incase your issue is related to ListSFTP

Jagapriyan · ‎11-08-2022

Thanks @MattWho This has solved the problem.

Jagapriyan · ‎11-07-2022

Hi @ckumar Yes I have verified. The port is not being used. However nifi throws the error.

Jagapriyan · ‎11-07-2022

I need to monitor the flowfiles that are queued for more than a day in NiFi. Is there a way to monitor that automatically? I have about 1400 processors running in production. It is very tedious to monitor it manually.

Jagapriyan · ‎11-07-2022

I'm trying to run listenHTTP processor for listening to a webhook notification. I'm relatively new to this processor. When I initialize, I observe java.net.bind exception. Address already in use. nifi url: http://localhost:8080/nifi/ (it has a public IP but we use NATing) Existing configuration Error Observed

Jagapriyan · ‎11-07-2022

Hi @MattWho Please find my responses - Does your target SFTP path have multiple sub-directories which are being searched? Is Search Recursively set to "true"? --> Search recursively is set to true - Are there symlink directories in use? - No - Have you looked the state recorded timestamp for your SFTP server directories? Are your missed files having older timestamps? -- Missing files have older time stamps. - How many files average are being written to the target SFTP between 12am and 1am each day? file count ranges from 10 - 100 and all these files are not being picked. Why not just have your listSFTP processor run all the time? Is this because timestamps are not being updated consistently? Even running the processor all the time with cron schedule is not picking the files.

Jagapriyan · ‎10-28-2022

Problem: I have set up a listSFTP processor to run between 2am to 2pm everyday. But the files are not being picked up by the processor. Existing setup and configuration: The files are generated at the SFTP server at 12 am to 1am every day. ListSFTP Configuration Schedule: listing strategy: tracking timestamps other tracking configuration When I start this processor, it runs as expected for 1 day or 2 and after that, the files are not picked. Is it the tracking time window of 3 hours that is affecting the file being listed?

Jagapriyan · ‎10-28-2022

Thank you @Matt

Jagapriyan · ‎10-13-2022

Hi @MattWho Thanks for the time to respond. How do you know when you have all the files for a given state? Usually, the files are status reports for the previous report. They are generated between 8am to 9am everyday. For example: If I have 2 countries US and Canada, there will be 90 files for US and 100 files for Canada. (the number of files may vary based on usage the date before) The flow: We have a list SFTP processor and Fetch SFTP processor that are collecting these files and storing on GCP. The requirement: I need to send an email notifying that files are delivered. What I Tried: Since I need to send one email for each country, I tried "replacetext" processor to remove the contents of the flow file and then merge the files into one. Since I can't use a hardcoded number, I couldn't give a value in minimum entries. My current Merge content processor

Jagapriyan · ‎09-28-2022

I have a requirement to notify once the files on the destination is processed. The pipeline is as follows: ListSFTP --> Update attribute (adds specific attribute for each state/province) ---> Send to Cloud Storage. Each state has a different count of files. Once the files are sent to cloud storage, notify the end user using email for each state. I have tried merging files based on a common attribute but, I need single files for each state so that I can send one notification for each state.

Jagapriyan · ‎08-23-2022

I have installed latest version of Java and NiFi. Now it works. Thanks for the suggestion.

Jagapriyan · ‎01-24-2022

This is the error im observing ERROR [main] org.apache.nifi.NiFi Failure to launch NiFi due to org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] Unable to make protected final java.lang.Class java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain) throws java.lang.ClassFormatError accessible: module java.base does not "opens java.lang" to unnamed module @4ec616d6 Got this error

Jagapriyan · ‎11-25-2021

Im trying to run Apache NiFi on my laptop. I can start the run-nifi.bat But when I try to access http://localhost:8443/. "This page isn’t working localhost didn’t send any data. ERR_EMPTY_RESPONSE" On the same laptop, I'm accessing different NiFi instances running on AWS. With different ports (Production). System: Windows10 Please help to connect

Online	Offline
Last Visited	‎03-29-2023 06:53 AM

Date Registered	‎11-25-2021 12:44 AM
Last Visited	‎03-29-2023 06:53 AM
Total Messages Posted	16

NiFi: Convert Avro to XML and replace first two li...

Compress Content in S3 bucket and Delete Original ...

Re: NiFi corn driven issue

Re: NiFi: Cron Schedule not working as expected

Re: Apache Nifi: ListenHTTP - Address already in u...

NiFI: Flowfile queue monitoring

Apache Nifi: ListenHTTP - Address already in use E...

Re: NiFi: Cron Schedule not working as expected

NiFi: Cron Schedule not working as expected

Re: NiFi: Merge files based on attribute and send ...

Re: NiFi: Merge files based on attribute and send ...

NiFi: Merge files based on attribute and send emai...

Re: Can Launch Apache NiFi on Windows Laptop

Re: Can Launch Apache NiFi on Windows Laptop

Can Launch Apache NiFi on Windows Laptop