Member since
07-30-2019
2909
Posts
1443
Kudos Received
846
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
30 | 04-23-2024 05:56 AM | |
21 | 04-22-2024 06:13 AM | |
144 | 04-17-2024 11:30 AM | |
111 | 04-16-2024 05:36 AM | |
74 | 04-15-2024 05:31 AM |
04-16-2024
05:36 AM
@whoknows The Spring Framework has been upgraded to 5.3.34 as part of the Apache NiFi 1.26 release as outlined in the following jira: https://issues.apache.org/jira/browse/NIFI-13037 My guess here is that you ran into dependency issues on startup after modifying those individual 1.25 nars? I suggest upgrading to 1.26 instead when it is released. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
04-15-2024
09:52 AM
@tmarkfeld The NiFi UI will not be avialable until you see the following lines in the nifi-app.log: 2024-03-17 16:39:59,500 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:
2024-03-17 16:39:59,500 INFO [main] org.apache.nifi.web.server.JettyServer https://localhost:8443/nifi Up until this time NiFi is still loading. I see you mentioned it stalls at unpacking the nars but eventually completes as you also said eventually you see NiFi listening on port 8443. You can use developer tools within your browser to see what calls have been made when you try to access your NiFi, you could then share what calls were made and which it appears to be hanging on. Hope this helps, Matt
... View more
04-15-2024
09:34 AM
1 Kudo
@Ytch All components on the NiFi canvas are executed as the NiFi service user and not as the user currently authenticated into the NiFi service. So what you should do is from each host in your NiFi cluster (do on every host since any one of the hosts can be elected as the primary node at any given time), open a command prompt window/console window, become the user that owns the NiFi process, and manually ssh/sftp to the target SFTP server. You will likely be prompted to add the target SFTP server to your known_hosts file for the NiFi service user. NiFi SFTP processor has no way of doing this interactive step. After successfully adding the SFTP to the known_hosts file for the NiFi service user, go back and try to start the GetSFTP or ListSFTP processors again to see if your issue is resolved. If not, please share your GetSFTP and ListSFTP processor component configurations. Also check the nifi-app.log for any exceptions or log output related to these processors. If no log output, you could also try enabling debug in the NiFi logback.xml for these processor classes to see what additional log output may be produced that could be useful here. classes for these processors are: org.apache.nifi.processors.standard.GetSFTP
org.apache.nifi.processors.standard.ListSFTP new log lines would look like this that you would add to logback.xml: <logger name="org.apache.nifi.processors.standard.GetSFTP" level="DEBUG"/>
<logger name="org.apache.nifi.processors.standard.ListSFTP" level="DEBUG"/> Simply add them in logback.xml where you see similar lines already. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
04-15-2024
06:09 AM
1 Kudo
@double_w NiFi state configuration is setup in the state-management.xml file located by default in NiFi's conf directory. When you say that state is being lost, are you specifically having issues with missing "cluster" state, "local" state, or both? Is your NiFi clustered or a standalone (non cluster) deployment? Default cluster state is stored in zookeeper (are you using an external zookeeper or the embedded zookeeper?). Any chance that your NiFi-2.0 is configured to use the new "kubernetes-provider" for cluster state management? If so, did you configure the "nifi.state.management.provider.cluster.previous" property in nifi.properties with the previous "zk-provider", so that on first startup, it migrates your cluster state from zk-provider to the empty kubernetes-provider? Default local-provider stores state in ./state/local. Have you compared the state-management.xml file configurations between both environments? Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
04-15-2024
05:31 AM
@EddyChan NiFi should only be generating a keystore and truststore on startup if you have not manually configured NiFi's nifi.properties file to use your personally generated keystore and truststore files. Even if they are generated, NiFi would still use your configured keystore and truststore files. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
04-03-2024
11:31 AM
3 Kudos
@Ytch For the "hostname" property you only provide the sftp server's fully qualified hostname. This hostname needs to be resolvable by the NiFi host. What version of Apache NiFi are you using? What errors or exceptions do you see when you start the processor? Have you tried using the listSFTP processor instead? Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
04-03-2024
11:24 AM
@s198 The hdfs.count.files flowfile attribute is added by the GetHDFSFileInfo processor and is added as a FlowFile attribute on to the FlowFile. In order for a processor to utilize a FlowFile Attribute, the processor property must support NiFi Expression Language (NEL). The "Max num entries" property on MergeContent processor does not support expression language. It would be difficult to support NiFi Expression Language in such a processor by nature of its design. But you could set the "correlation Attribute name" property to an attribute that all split FlowFiles will have same value set like "hdfs.path". That would put all FlowFiles with sam value in this FlowFile attribute in the same bin. This would allow you to process multiple different HDFS directory file merges concurrently. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
04-03-2024
05:59 AM
1 Kudo
@Adhitya The more detail you can provide the better assistance we may be able to provide here. 1. Can you share the exact HDF, CFM, or Apache NiFi version? 2. Have you searched your NiFi flow configuration History from the NiFi UI --> global menu for that component id to see what history exists for its creation and removal? 3. Have you ever manually edited the flow.xml.gz of flow.json.gz files? While I can not explain from what has been shared how you got in to this state, there are steps to get out of it. When NiFi is started, it loads the dataflow(s) from the flow.xml.gz or flow.json.gz depending on your NiFi version. (Basically, if the flow.json.gz exists, then NiFi is using it during startup even though NiFi also writes out the legacy flow.xml.gz still). So you could shutdown your NiFi and search the flow.xml.gz/flow.json.gz for all occurrences of the ID of the component from shared exception just in case there are other components and not just this DistributedMapCacheClientService still referencing it. Then delete that reference ID. I recommend making a backup of the flow.xml.gz and flow.json.gz before any attempts to manually edit these files. Then start Your NiFi using this modified flow.xml.gz/flow.json.gz file. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
04-02-2024
01:22 PM
@edim2525 The entire datflow(s) reside in NiFi heap memory and are persisted to disk in the flow.json.gz file. This includes the current set state for each component. Every time a change is made to the NiFi canvas, the current persisted flow.json.gz is moved to archive and a new flow.json.gz is written. When NiFi is started it will load the flow.json.gz in to NiFi heap memory and set the state to the state for each component recorded in the the flow.json.gz that is loaded. Only time this is not true is when the nifi.properties property "nifi.flowcontroller.autoResumeState" has been set to false. When set to false, all "RUNNING" components set in the flow.josn.gz will load into heap as "STOPPED". NiFi will then archive the current flow.json.gz and write a new flow.json.gz with those new "STOPPED" states. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
04-02-2024
06:16 AM
@EddyChan The out-of-box Apache NiFi self-signed certificate generation was added to make it easy for first time users to experiment with a secure NiFi instance. Just like the Single user authentication and and single user authorizer, these were not intended to be used for long term or production use cases. There is no configuration option to extend the lifetime. For long term use or production setups, you should be generating your own signed certificates for use in your NiFi (preferable signed by a trusted authority rather then being self-signed). You could use the NiFi TLS toolkit still available in the Apache NiFi 1.x releases to generate your own certificates for keystore and truststore. You could generate your own following guidelines for NiFi certificates: Security Configuration You could use a free online service to generate certificates. Please help our community grow. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more