Cloudera Community
Announcements
We’ve updated our product names and community labels - click here for full details
Jagapriyan
user rank
Contributor

Fork Record Processor cannot handle Large files

by Contributor in Support Questions
‎10-08-2024 11:25 PM
1 Kudo
‎10-08-2024 11:25 PM
1 Kudo
I'm using ForkRecord processor in Apache NiFi for a use case where I need to retain the parent of the record and it is working smoothly for file sizes up to 100 MB. When I pass larger files say for example files with 250 MB or above it is taking longer time resulting in huge backlogs of files.  Is there any alternate flow I can build to avoid such backlogs? Can I completely avoid fork record processor and choose some other flow logic to solve this ?   @mburgess @MattWho   Please help here ... View more
Labels:

MiniFi - nifi.sensitive.props.key is not set

by Contributor in Support Questions
‎05-23-2024 06:04 AM
‎05-23-2024 06:04 AM
I'm trying to run  miniFi  in ubuntu. When I try to start miniFi, I got following error in bootstrap and app log.  App.log 2024-05-23 08:49:45,416 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 22, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected. 2024-05-23 08:49:45,439 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 99, col 28: cvc-complex-type.2.4.a: Invalid content was found starting with element 'networkInterface'. One of '{inputPort, outputPort}' is expected. 2024-05-23 08:49:45,833 INFO [main] o.a.nifi.headless.HeadlessNiFiServer Loading Flow... 2024-05-23 08:49:45,864 WARN [main] o.a.nifi.headless.HeadlessNiFiServer Failed to start headless server... shutting down. java.lang.Exception: Unable to load flow due to: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:176) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:163) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:64) at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:265) Caused by: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required at org.apache.nifi.encrypt.PropertyEncryptorFactory.getPropertyEncryptor(PropertyEncryptorFactory.java:42) at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:129) ... 3 common frames omitted 2024-05-23 08:49:45,864 INFO [Thread-0] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... 2024-05-23 08:49:45,883 WARN [Thread-0] o.a.nifi.headless.HeadlessNiFiServer Problem occurred ensuring flow controller or repository was properly terminated due to java.lang.NullPointerException 2024-05-23 08:49:45,896 ERROR [Thread-0] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi 2024-05-23 08:49:45,896 INFO [Thread-0] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise).   Bootstrap  log   2024-05-23 08:49:45,416 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 22, col 29: cvc-complex-type.2.4.a: Invalid content was found starting with element 'maxConcurrentTasks'. One of '{bundle}' is expected. 2024-05-23 08:49:45,439 WARN [main] org.apache.nifi.util.FlowParser Schema validation error parsing Flow Configuration at line 99, col 28: cvc-complex-type.2.4.a: Invalid content was found starting with element 'networkInterface'. One of '{inputPort, outputPort}' is expected. 2024-05-23 08:49:45,833 INFO [main] o.a.nifi.headless.HeadlessNiFiServer Loading Flow... 2024-05-23 08:49:45,864 WARN [main] o.a.nifi.headless.HeadlessNiFiServer Failed to start headless server... shutting down. java.lang.Exception: Unable to load flow due to: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:176) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:163) at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:64) at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:265) Caused by: java.lang.IllegalArgumentException: NiFi Sensitive Properties Key [nifi.sensitive.props.key] is required at org.apache.nifi.encrypt.PropertyEncryptorFactory.getPropertyEncryptor(PropertyEncryptorFactory.java:42) at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:129) ... 3 common frames omitted 2024-05-23 08:49:45,864 INFO [Thread-0] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... 2024-05-23 08:49:45,883 WARN [Thread-0] o.a.nifi.headless.HeadlessNiFiServer Problem occurred ensuring flow controller or repository was properly terminated due to java.lang.NullPointerException 2024-05-23 08:49:45,896 ERROR [Thread-0] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi 2024-05-23 08:49:45,896 INFO [Thread-0] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise).   ... View more
Labels:

Re: NiFi: Cron Schedule not working as expected

by Contributor in Support Questions
‎11-08-2022 10:48 PM
‎11-08-2022 10:48 PM
Thanks @MattWho  This has solved the problem.  ... View more

Re: Apache Nifi: ListenHTTP - Address already in u...

by Contributor in Support Questions
‎11-07-2022 04:47 AM
‎11-07-2022 04:47 AM
Hi @ckumar  Yes I have verified. The port is not being used. However nifi throws the error.  ... View more

Apache Nifi: ListenHTTP - Address already in use E...

by Contributor in Support Questions
‎11-07-2022 01:20 AM
‎11-07-2022 01:20 AM
I'm trying to run listenHTTP processor for listening to a webhook notification.  I'm relatively new to this processor. When I initialize, I observe java.net.bind exception. Address already in use.    nifi url: http://localhost:8080/nifi/    (it has a public IP but we use NATing)    Existing configuration         Error Observed         ... View more
Labels:

Re: NiFi: Cron Schedule not working as expected

by Contributor in Support Questions
‎11-07-2022 01:03 AM
‎11-07-2022 01:03 AM
Hi @MattWho  Please find my responses - Does your target SFTP path have multiple sub-directories which are being searched?  Is Search Recursively set to "true"?   --> Search recursively is set to true - Are there symlink directories in use? - No - Have you looked the state recorded timestamp for your SFTP server directories?  Are your missed files having older timestamps? -- Missing files have older time stamps.  - How many files average are being written to the target SFTP between 12am and 1am each day? file count ranges from 10 - 100 and all these files are not being picked.    Why not just have your listSFTP processor run all the time?  Is this because timestamps are not being updated consistently? Even running the processor all the time with cron schedule is not picking the files.  ... View more

NiFi: Cron Schedule not working as expected

by Contributor in Support Questions
‎10-28-2022 12:39 AM
‎10-28-2022 12:39 AM
Problem: I have set up a listSFTP processor to run between 2am to 2pm everyday.  But the files are not being picked up by the processor.    Existing setup and configuration: The files are generated at the SFTP server at 12 am to 1am every day.  ListSFTP Configuration Schedule:   listing strategy: tracking timestamps other tracking configuration     When I start this processor, it runs as expected for 1 day or 2 and after that, the files are not picked.  Is it the tracking time window of 3 hours that is affecting the file being listed?   ... View more
Labels:

Re: NiFi: Merge files based on attribute and send ...

by Contributor in Support Questions
‎10-28-2022 12:05 AM
‎10-28-2022 12:05 AM
Thank you @Matt  ... View more

Re: NiFi: Merge files based on attribute and send ...

by Contributor in Support Questions
‎10-13-2022 05:41 AM
‎10-13-2022 05:41 AM
Hi @MattWho  Thanks for the time to respond.  How do you know when you have all the files for a given state?  Usually, the files are status reports for the previous report. They are generated between 8am to 9am everyday.  For example: If I have 2 countries US and Canada, there will be 90 files for US and 100 files for Canada. (the number of files may vary based on usage the date before) The flow: We have a list SFTP processor and Fetch SFTP processor that are collecting these files and storing on GCP. The requirement:  I need to send an email notifying that files are delivered.  What I Tried:  Since I need to send one email for each country, I tried  "replacetext" processor to remove the contents of the flow file and then merge the files into one. Since I can't use a hardcoded number, I couldn't give a value in minimum entries.  My current Merge content processor     ... View more

NiFi: Merge files based on attribute and send emai...

by Contributor in Support Questions
‎09-28-2022 05:02 AM
‎09-28-2022 05:02 AM
I have a requirement to notify once the files on the destination is processed.  The pipeline is as follows: ListSFTP --> Update attribute (adds specific attribute for each state/province) ---> Send to Cloud Storage.  Each state has a different count of files.  Once the files are sent to cloud storage, notify the end user using email for each state.  I have tried merging files based on a common attribute but, I need single files for each state so that I can send one notification for each state.    ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎11-05-2024 10:28 AM
Community Statistics
Member Since ‎11-25-2021 12:44 AM
Last Visited ‎11-05-2024 10:28 AM
Posts 19
Kudos received 2