About Jagapriyan

Jagapriyan · ‎10-08-2024

I'm using ForkRecord processor in Apache NiFi for a use case where I need to retain the parent of the record and it is working smoothly for file sizes up to 100 MB. When I pass larger files say for example files with 250 MB or above it is taking longer time resulting in huge backlogs of files. Is there any alternate flow I can build to avoid such backlogs? Can I completely avoid fork record processor and choose some other flow logic to solve this ? @mburgess @MattWho Please help here

Jagapriyan · ‎05-23-2024

I'm trying to run miniFi in ubuntu. When I try to start miniFi, I got following error in bootstrap and app log. App.log 2024-05-23 08:49:45,416 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 22, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected. 2024-05-23 08:49:45,439 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 99, col 28: cvc-complex-type.2.4.a: Invalid content was found starting with element 'networkInterface'. One of '{inputPort, outputPort}' is expected. 2024-05-23 08:49:45,833 INFO [main] o.a.nifi.headless.HeadlessNiFiServer Loading Flow... 2024-05-23 08:49:45,864 WARN [main] o.a.nifi.headless.HeadlessNiFiServer Failed to start headless server... shutting down. java.lang.Exception: Unable to load flow due to: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:176) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:163) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:64) at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:265) Caused by: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required at org.apache.nifi.encrypt.PropertyEncryptorFactory.getPropertyEncryptor(PropertyEncryptorFactory.java:42) at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:129) ... 3 common frames omitted 2024-05-23 08:49:45,864 INFO [Thread-0] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... 2024-05-23 08:49:45,883 WARN [Thread-0] o.a.nifi.headless.HeadlessNiFiServer Problem occurred ensuring flow controller or repository was properly terminated due to java.lang.NullPointerException 2024-05-23 08:49:45,896 ERROR [Thread-0] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi 2024-05-23 08:49:45,896 INFO [Thread-0] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise). Bootstrap log 2024-05-23 08:49:45,416 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 22, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected. 2024-05-23 08:49:45,439 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 99, col 28: cvc-complex-type.2.4.a: Invalid content was found starting with element 'networkInterface'. One of '{inputPort, outputPort}' is expected. 2024-05-23 08:49:45,833 INFO [main] o.a.nifi.headless.HeadlessNiFiServer Loading Flow... 2024-05-23 08:49:45,864 WARN [main] o.a.nifi.headless.HeadlessNiFiServer Failed to start headless server... shutting down. java.lang.Exception: Unable to load flow due to: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:176) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:163) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:64) at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:265) Caused by: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required at org.apache.nifi.encrypt.PropertyEncryptorFactory.getPropertyEncryptor(PropertyEncryptorFactory.java:42) at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:129) ... 3 common frames omitted 2024-05-23 08:49:45,864 INFO [Thread-0] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... 2024-05-23 08:49:45,883 WARN [Thread-0] o.a.nifi.headless.HeadlessNiFiServer Problem occurred ensuring flow controller or repository was properly terminated due to java.lang.NullPointerException 2024-05-23 08:49:45,896 ERROR [Thread-0] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi 2024-05-23 08:49:45,896 INFO [Thread-0] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise).

Jagapriyan · ‎04-16-2023

Due to space constraints, it is not bundled by default. You can however use it. Follow the below link https://jd-bots.com/2021/09/10/fixed-inferavroschema-not-found-in-latest-version-of-nifi-1-10-0-and-above/

Jagapriyan · ‎03-27-2023

I have a requirement where I need to execute a SQL query to a DB and transform the returned avro file to XML. First I need to replace a few of the records in the file. So I used a split record processor with Avro reader and XML writer to split the records Two replace text processors to replace the required texts in each record Then merge the files into a single file Once the file is merged, I need to replace the XML declaration and root recrod of XML with custom values. The resultant file is about 2GB in size. Can I use sed 1,2d command in execute stream command processor to remove the files and replace text processor to prepend the required text. If I use this method, will the large file occupy system memory and affect performance of other pipelines? Flow is attached

Jagapriyan · ‎02-20-2023

I have a setup where there is 1 S3 bucket for each day and 4k files per bucket. The requirement I have is 1) Compress all the files 2) archive the files 3) push the files to different bucket 4) delete the source files after the success of the above three operations. Please suggest a nifi flow for this requirement. What I have tried: I tried listing the files using ListS3, Fetching the files using FetchS3, Compress content and tar them using merge content processor. After merge I could not get the filenames to delete from source bucket.

Jagapriyan · ‎11-17-2022

Hi @hargav I had similar issues as well. If you could share your configuration, I can help out Initially, check what is your listing strategy, have you turned on recursive searching, target system timestamp precision etc., incase your issue is related to ListSFTP

Jagapriyan · ‎11-08-2022

Thanks @MattWho This has solved the problem.

Jagapriyan · ‎11-07-2022

Hi @ckumar Yes I have verified. The port is not being used. However nifi throws the error.

Jagapriyan · ‎11-07-2022

I need to monitor the flowfiles that are queued for more than a day in NiFi. Is there a way to monitor that automatically? I have about 1400 processors running in production. It is very tedious to monitor it manually.

Jagapriyan · ‎11-07-2022

I'm trying to run listenHTTP processor for listening to a webhook notification. I'm relatively new to this processor. When I initialize, I observe java.net.bind exception. Address already in use. nifi url: http://localhost:8080/nifi/ (it has a public IP but we use NATing) Existing configuration Error Observed

Online	Offline
Last Visited	‎11-05-2024 10:28 AM

Member Since	‎11-25-2021 12:44 AM
Last Visited	‎11-05-2024 10:28 AM
Posts	19
Kudos received	2

Cloudera Community

Fork Record Processor cannot handle Large files

MiniFi - nifi.sensitive.props.key is not set

Re: Did InferAvroSchema get removed?

NiFi: Convert Avro to XML and replace first two li...

Compress Content in S3 bucket and Delete Original ...

Re: NiFi corn driven issue

Re: NiFi: Cron Schedule not working as expected

Re: Apache Nifi: ListenHTTP - Address already in u...

NiFI: Flowfile queue monitoring

Apache Nifi: ListenHTTP - Address already in use E...