- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Unable to start MiNiFi because of an error caused due to the number of threads
- Labels:
-
Apache MiNiFi
-
Apache NiFi
Created 09-09-2024 03:50 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
What am I doing:
I am new to minifi and I want to get a bit workout on how to use minifi and generate a dataflow and send it to Apache NiFi.
After going through the web I understood that first we need to design the flow using the NiFi and download its JSON definition as json.gz format and place it in the conf folder of minifi. Then start the minifi using run-minifi.bat file.
- These are things I have done to build the dataflow for MiNiFi:
- First I have created a pipeline to read from a table present in my Postgres SQL database using a ExecuteSQL processor, Remote Process Group, Input Port and logAttribute Processor.
Input port connected to a LogAttribute to receive the data from the Remote Process Group - Second, I have downloaded the ExecuteSQL, Remote Process Group and LogAttribute pipeline as a JSON file and compressed it as flow.json.gz and placed it in the conf folder and started the minifi instance.
- First I have created a pipeline to read from a table present in my Postgres SQL database using a ExecuteSQL processor, Remote Process Group, Input Port and logAttribute Processor.
The Error I am Facing:
java.lang.Exception: Unable to load flow due to: org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.IllegalArgumentException: Cannot set max number of threads to less than 1
at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:194)
at org.apache.nifi.minifi.StandardMiNiFiServer.start(StandardMiNiFiServer.java:62)
at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:141)
at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:62)
at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:226)
Caused by: org.apache.nifi.controller.serialization.FlowSynchronizationException: java.lang.IllegalArgumentException: Cannot set max number of threads to less than 1
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:477)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.sync(VersionedFlowSynchronizer.java:224)
at org.apache.nifi.controller.FlowController.synchronize(FlowController.java:1729)
at org.apache.nifi.persistence.StandardFlowConfigurationDAO.load(StandardFlowConfigurationDAO.java:91)
at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:805)
at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:526)
at org.apache.nifi.headless.HeadlessNiFiServer.start(HeadlessNiFiServer.java:170)
... 4 common frames omitted
Caused by: java.lang.IllegalArgumentException: Cannot set max number of threads to less than 1
at org.apache.nifi.controller.FlowController.setMaxThreadCount(FlowController.java:1783)
at org.apache.nifi.controller.FlowController.setMaxTimerDrivenThreadCount(FlowController.java:1766)
at org.apache.nifi.controller.serialization.VersionedFlowSynchronizer.synchronizeFlow(VersionedFlowSynchronizer.java:410)
... 10 common frames omitted
What did I try to resolve the Error:
- I tried adding this property in my minifi.properties but the file is getting reset to the previous properties making this to get erased on every start of minifi instance nifi.flow.engine.threads= 5 .
- I tried adding this in my bootstrap.conf file but the error still persists. nifi.minifi.flow.max.concurrent.threads=1.
I'm hoping someone who has encountered this error before or has an idea about how to resolve it can help me please. Thank you in advance.
Created 09-09-2024 04:42 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Leo3103, Welcome to our community! To help you get the best possible answer, I have tagged in our NiFi experts @MattWho @SAMSAL who may be able to assist you further.
Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.
Regards,
Vidya Sargur,Community Manager
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
Created 09-09-2024 05:37 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Leo3103
It appeasr you are handling the conversion of your NiFi flow Definition for use in your MiNiFi incorrectly as per doscumetation:
https://github.com/apache/nifi/blob/main/minifi/minifi-docs/src/main/markdown/minifi-java-agent-quic...
You should be downloading your flow definition (json file) via the NiFi UI.
Then you should be renaming that file "flow.json.raw" (no mention of compression here) and place it in the MiNiFi conf directory.
Once you have your flow.json.raw file in the minifi/conf directory, launch that instance of MiNiFi and your dataflow begins automatically.
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
Created on 09-09-2024 11:21 PM - edited 09-09-2024 11:24 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@VidyaSargur @MattWho Thank you for your time and extending your help. Hey Matt I did as you mentioned in the reply and followed the instructions in the quick start too. Now I am facing a new error
cannot invoke "org.apache.nifi.flow.VersionedProcessGroup.getIdentifier()" because "rootGroup" is null
here is the complete error too can you please help me if you have an idea on how to resolve this error.
Command Exception happened during MiNiFi startup
org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeException: Unable to perform reload of received configuration change
at org.apache.nifi.minifi.bootstrap.service.MiNiFiConfigurationChangeListener.handleChange(MiNiFiConfigurationChangeListener.java:103)
at org.apache.nifi.minifi.bootstrap.command.StartRunner.regenerateFlowConfiguration(StartRunner.java:279)
at org.apache.nifi.minifi.bootstrap.command.StartRunner.start(StartRunner.java:131)
at org.apache.nifi.minifi.bootstrap.command.StartRunner.runCommand(StartRunner.java:107)
at org.apache.nifi.minifi.bootstrap.RunMiNiFi.run(RunMiNiFi.java:133)
at org.apache.nifi.minifi.bootstrap.RunMiNiFi.main(RunMiNiFi.java:150)
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.nifi.flow.VersionedProcessGroup.getIdentifier()" because "rootGroup" is null
at org.apache.nifi.minifi.commons.service.StandardFlowEnrichService.enrichFlow(StandardFlowEnrichService.java:115)
at org.apache.nifi.minifi.bootstrap.service.MiNiFiConfigurationChangeListener.handleChange(MiNiFiConfigurationChangeListener.java:92)
... 5 common frames omitted
Here is my json definition can you please help me out to resolve this error.
Created 09-11-2024 02:03 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Leo3103
This exception might be caused by a missing nar file in your MiNiFi.
MiNiFi does not include all the NiFi nars.
https://github.com/apache/nifi/blob/main/minifi/minifi-docs/src/main/markdown/minifi-java-agent-quic...
A quick look through your flow definition I saw you were using the ExecuteSQL processor. While ExecuteSQL is part of one of the included nars, it does require another nar that is not included (this is noted in the above linked quick start guide).
You'll need to add the "nifi-dbcp-service-nar" from your NiFi distribution to your MiNiFi's lib directory.
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
Created 09-12-2024 12:31 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@MattWho thank you for responding back, I do understand I did not include the "nifi-dbcp-service-nar". But this time I am trying to run a new flow which is defined as below
When I downloaded the flow definition of the internalTest and placed it in the conf folder of minifi I am still facing the same issue.
Created 09-25-2024 05:34 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@MattWho I did try adding the "nifi-dbcp-service-nar" from NiFi distribution to MiNiFi's lib directory. Even then I am facing the same error
C:\SoftwareCenterApplications\minifi-2.0.0-M4-bin\minifi-2.0.0-M4\bin>run-minifi.bat
2024-09-25 18:00:58,785 INFO [main] o.a.n.minifi.properties.PropertiesLoader Loading Bootstrap Properties [conf\bootstrap.conf]
2024-09-25 18:00:58,911 INFO [main] o.a.n.minifi.properties.PropertiesLoader Loading Bootstrap Properties [conf\bootstrap.conf]
2024-09-25 18:00:58,917 INFO [main] o.a.n.minifi.properties.PropertiesLoader Loading Bootstrap Properties [conf\bootstrap.conf]
2024-09-25 18:00:59,047 INFO [main] o.a.n.minifi.properties.PropertiesLoader Loading Bootstrap Properties [conf\bootstrap.conf]
2024-09-25 18:00:59,612 ERROR [main] o.apache.nifi.minifi.bootstrap.Command Exception happened during MiNiFi startup
org.apache.nifi.minifi.bootstrap.configuration.ConfigurationChangeException: Unable to perform reload of received configuration change
at org.apache.nifi.minifi.bootstrap.service.MiNiFiConfigurationChangeListener.handleChange(MiNiFiConfigurationChangeListener.java:103)
at org.apache.nifi.minifi.bootstrap.command.StartRunner.regenerateFlowConfiguration(StartRunner.java:279)
at org.apache.nifi.minifi.bootstrap.command.StartRunner.start(StartRunner.java:131)
at org.apache.nifi.minifi.bootstrap.command.StartRunner.runCommand(StartRunner.java:107)
at org.apache.nifi.minifi.bootstrap.RunMiNiFi.run(RunMiNiFi.java:133)
at org.apache.nifi.minifi.bootstrap.RunMiNiFi.main(RunMiNiFi.java:150)
Caused by: java.lang.NullPointerException: Cannot invoke "org.apache.nifi.flow.VersionedProcessGroup.getIdentifier()" because "rootGroup" is null
at org.apache.nifi.minifi.commons.service.StandardFlowEnrichService.enrichFlow(StandardFlowEnrichService.java:115)
at org.apache.nifi.minifi.bootstrap.service.MiNiFiConfigurationChangeListener.handleChange(MiNiFiConfigurationChangeListener.java:92)
... 5 common frames omitted
Can you please help me out if you have any idea on this. Thank you in advance.
Created 09-30-2024 12:09 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Leo3103
Things to note.
- Your process group that you will be exporting a flow definition for can NOT contain local input or output ports at the top level. ---> The Process Group will become the "root" process group in your MiNiFi deployment. Only "remote" input or output ports are supported at the root level. Within the Process group you are exporting, you can have child process group deeper down that contain local input and output ports. But the test process group "internalTest" you are exporting has a local output port.
- If you take entire flow.json.gz from the NiFi conf directory and rename it to flow.json.raw, you can start MiNiFi with no issues.
- Since you are exporting a flow definition of a process group, you'll need to utilize the MiNiFi toolkit to transform it in to the proper format that can be loaded by MiNiFi. The MiNiFi-toolkit can be downloaded from here: https://nifi.apache.org/download/ (select "MINIFI' and click download link for Toolkit). Execute:
./minifi-toolkit/bin/config.sh transform-nifi <exported flow definition> flow.json.raw
Now edit the flow.json.raw file and edit the following property at start of file (value can not be 0.)
"maxTimerDrivenThreadCount":5
- Now you can start your MiNiFi and it will create the flow.json.gz as it starts.
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
Created 09-30-2024 12:21 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Leo3103
Here are some additional challenges you may have:
- NiFi flow definitions do not export with any encrypted values. So it your dataflow uses a component that has sensitive properties (passwords), those will not exist in the flow.json you export. You could get the encrypted values ( they will look like enc{....<long string>....}) from the NiFi flow.json.gz file and add them in to your flow.json.raw produced by the toolkit.
- In order for your MiNiFi to be able to load a flow.json.raw containing sensitive encrypted values, the same sensitive.props.key value and sensitive.props.algorithm used in your NiFi (nifi.properties) must be used in your MiNiFi bootstrap.conf.
Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.
Thank you,
Matt
Created on 10-08-2024 03:51 AM - edited 10-08-2024 03:53 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @MattWho, once again Thank you as always.
After going through your answer these are things I did.
1) I have created a new flow as attached below
Where Image 1 represents the complete flow and Image 2 shows the flow inside the remoteMiNiFiTest.
2) After designing the flow I have download the flow of the remoteMiNiFiTest as a .json file and converted into flow.json.gz using the minifi toolkit. ( Why I didn't convert it into a flow.json.raw
(2.a) Because I am facing the same error as mentioned in the above thread
"org.apache.nifi.flow.VersionedProcessGroup.getIdentifier()" because "rootGroup" is null
(2.b) While running the run-minifi.bat file the instance of minifi is not starting up and prompting that the format of the flow configuration file is not in the format of GZIP ).
3) So in - order to tackle these errors I have converted it into flow.json.gz rather flow.json.raw.
4) Even though after my minifi is getting started up without any issues or breakdowns due to exception thrown by errors, the flowfile data is not getting generated by minifi.
Can you please help me out if you know what all wrong I am doing if you have an idea or so. Thank you for responding for all the previous troubles.