About Jagapriyan

Jagapriyan · ‎10-08-2024

I'm using ForkRecord processor in Apache NiFi for a use case where I need to retain the parent of the record and it is working smoothly for file sizes up to 100 MB. When I pass larger files say for example files with 250 MB or above it is taking longer time resulting in huge backlogs of files. Is there any alternate flow I can build to avoid such backlogs? Can I completely avoid fork record processor and choose some other flow logic to solve this ? @mburgess @MattWho Please help here

Jagapriyan · ‎05-23-2024

I'm trying to run miniFi in ubuntu. When I try to start miniFi, I got following error in bootstrap and app log. App.log 2024-05-23 08:49:45,416 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 22, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected. 2024-05-23 08:49:45,439 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 99, col 28: cvc-complex-type.2.4.a: Invalid content was found starting with element 'networkInterface'. One of '{inputPort, outputPort}' is expected. 2024-05-23 08:49:45,833 INFO [main] o.a.nifi.headless.HeadlessNiFiServer Loading Flow... 2024-05-23 08:49:45,864 WARN [main] o.a.nifi.headless.HeadlessNiFiServer Failed to start headless server... shutting down. java.lang.Exception: Unable to load flow due to: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:176) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:163) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:64) at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:265) Caused by: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required at org.apache.nifi.encrypt.PropertyEncryptorFactory.getPropertyEncryptor(PropertyEncryptorFactory.java:42) at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:129) ... 3 common frames omitted 2024-05-23 08:49:45,864 INFO [Thread-0] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... 2024-05-23 08:49:45,883 WARN [Thread-0] o.a.nifi.headless.HeadlessNiFiServer Problem occurred ensuring flow controller or repository was properly terminated due to java.lang.NullPointerException 2024-05-23 08:49:45,896 ERROR [Thread-0] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi 2024-05-23 08:49:45,896 INFO [Thread-0] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise). Bootstrap log 2024-05-23 08:49:45,416 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 22, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected. 2024-05-23 08:49:45,439 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 99, col 28: cvc-complex-type.2.4.a: Invalid content was found starting with element 'networkInterface'. One of '{inputPort, outputPort}' is expected. 2024-05-23 08:49:45,833 INFO [main] o.a.nifi.headless.HeadlessNiFiServer Loading Flow... 2024-05-23 08:49:45,864 WARN [main] o.a.nifi.headless.HeadlessNiFiServer Failed to start headless server... shutting down. java.lang.Exception: Unable to load flow due to: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:176) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:163) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:64) at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:265) Caused by: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required at org.apache.nifi.encrypt.PropertyEncryptorFactory.getPropertyEncryptor(PropertyEncryptorFactory.java:42) at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:129) ... 3 common frames omitted 2024-05-23 08:49:45,864 INFO [Thread-0] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... 2024-05-23 08:49:45,883 WARN [Thread-0] o.a.nifi.headless.HeadlessNiFiServer Problem occurred ensuring flow controller or repository was properly terminated due to java.lang.NullPointerException 2024-05-23 08:49:45,896 ERROR [Thread-0] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi 2024-05-23 08:49:45,896 INFO [Thread-0] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise).

Jagapriyan · ‎04-16-2023

Due to space constraints, it is not bundled by default. You can however use it. Follow the below link https://jd-bots.com/2021/09/10/fixed-inferavroschema-not-found-in-latest-version-of-nifi-1-10-0-and-above/

Jagapriyan · ‎11-17-2022

Hi @hargav I had similar issues as well. If you could share your configuration, I can help out Initially, check what is your listing strategy, have you turned on recursive searching, target system timestamp precision etc., incase your issue is related to ListSFTP

Jagapriyan · ‎11-08-2022

Thanks @MattWho This has solved the problem.

Jagapriyan · ‎11-07-2022

Hi @ckumar Yes I have verified. The port is not being used. However nifi throws the error.

Jagapriyan · ‎11-07-2022

I'm trying to run listenHTTP processor for listening to a webhook notification. I'm relatively new to this processor. When I initialize, I observe java.net.bind exception. Address already in use. nifi url: http://localhost:8080/nifi/ (it has a public IP but we use NATing) Existing configuration Error Observed

Jagapriyan · ‎11-07-2022

Hi @MattWho Please find my responses - Does your target SFTP path have multiple sub-directories which are being searched? Is Search Recursively set to "true"? --> Search recursively is set to true - Are there symlink directories in use? - No - Have you looked the state recorded timestamp for your SFTP server directories? Are your missed files having older timestamps? -- Missing files have older time stamps. - How many files average are being written to the target SFTP between 12am and 1am each day? file count ranges from 10 - 100 and all these files are not being picked. Why not just have your listSFTP processor run all the time? Is this because timestamps are not being updated consistently? Even running the processor all the time with cron schedule is not picking the files.

Jagapriyan · ‎10-28-2022

Problem: I have set up a listSFTP processor to run between 2am to 2pm everyday. But the files are not being picked up by the processor. Existing setup and configuration: The files are generated at the SFTP server at 12 am to 1am every day. ListSFTP Configuration Schedule: listing strategy: tracking timestamps other tracking configuration When I start this processor, it runs as expected for 1 day or 2 and after that, the files are not picked. Is it the tracking time window of 3 hours that is affecting the file being listed?

Jagapriyan · ‎10-28-2022

Thank you @Matt

Online	Offline
Last Visited	‎11-05-2024 10:28 AM

Member Since	‎11-25-2021 12:44 AM
Last Visited	‎11-05-2024 10:28 AM
Posts	19
Kudos received	2

Cloudera Community

Fork Record Processor cannot handle Large files

MiniFi - nifi.sensitive.props.key is not set

Re: Did InferAvroSchema get removed?

Re: NiFi corn driven issue

Re: NiFi: Cron Schedule not working as expected

Re: Apache Nifi: ListenHTTP - Address already in u...

Apache Nifi: ListenHTTP - Address already in use E...

Re: NiFi: Cron Schedule not working as expected

NiFi: Cron Schedule not working as expected

Re: NiFi: Merge files based on attribute and send ...