Member since
07-30-2019
2922
Posts
1451
Kudos Received
850
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
136 | 05-06-2024 10:40 AM | |
78 | 05-03-2024 08:41 AM | |
192 | 04-26-2024 06:40 AM | |
249 | 04-25-2024 06:16 AM | |
605 | 04-23-2024 05:56 AM |
05-06-2024
10:40 AM
@jonay__reyes or @DeepakDonde The issue you are encountering if caused by a code change so the the invokeHTTP would encode URLs automatically. This issue is triggered if your URL is already encoded. The URL encoding change will convert all '%' to '%25'. Workarounds/solutions: Remove your url encoding and allow processor to do that encoding. If your URL is not URL encoded already and happens to contain '%' in the URL you can do the following: If using Apache NiFi 1.25 verions: Upgrade to NIFi 1.26 which contains fix NIFI-12842. (now released) Try adding Apache NiFi 1.26 version of the NiFi standard nar to your 1.25 install. Downgrade to Apache NiFi 1.24 If using Apache NiFi 2.0.0-M2 Wait for upcoming release of NiFi 2.0.0-M3 and upgrade, which will contain fix. Downgrade to Apache NiFi 2.0.0-M1 Try adding Apache NiFi 2.0.0-M1 Standard nar to your 2.0.0-M2 install and switch to using older 2.0.0-M1 invokeHTTP processor. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
05-06-2024
10:14 AM
1 Kudo
@jonay__reyes Apache NiFi 1.x and NiFi 2.x are major release versions, I would not expect that you could mix component versioned nars successfully between those versions. My suggestion in the above post was you could try: 1. If using Apache NiFi 1.25, you could add the 1.24 nar. 2. If using Apache NiFi 2.0.0-M2, you could add the Apache NiFi 2.0.0.-M1 nar. The NiFi standard nar contains so many core libraries and components, I can't guarantee it will load successfully, but have done is successfully in the older releases of Apache NIFi 1.x versions. Never tried with Apache NiFi 2.x major milestone (M1, M2, ...) release versions. With above being said, Nothing you shared above appears to be an ERROR. So I am no clear in exactly what you mean when you say "doesn't seem to work". Did not startup? Could the UI be accessed? When adding a new component do you see multiple versions for the components included in the standard nar? When you load in to duplicate nars of different versions, NiFi does not change anything about the components loaded on the canvas. What you should see when dragging and adding new components to the canvas is multiple version options for the same component class to choose from. If you are still using the 2.0.0-M2 version of invokeHTTP, it is still going to have issues. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
05-03-2024
08:41 AM
1 Kudo
@manishg The elected cluster coordinator (elected by Zookeeper) is responsible for receiving and processing heartbeats from other nodes in the cluster. It handles the connecting, reconnecting, and manual disconnecting of NiFi nodes. The Cluster coordinator is also responsible for replicating user request to all nodes in the cluster and get confirmation from those nodes that the request was completed successfully. Assume a 3 node cluster with following: node1 - elected cluster coordinator node2 - elected primary node node3 Role of Cluster Coordinator: A user can access the NiFi cluster via any of the 3 node's URL. So lets say a user logs in node3's UI. When that user interacts with node 3 UI that request is proxied to the currently elected cluster coordinator node that in turn replicates the request all 3 nodes (example: add a processor, configure a processor, empty a queue, etc...). If one of the nodes were to fail to complete the request, that node would get disconnected. I may attempt to auto-reconnect later (In newer version of NiFi a connecting node can inherit the clusters flow and replace it local flow only if doing so would not result in dataloss. Role of the Primary Node: The elected primary node is responsible for scheduling the execution of any NiF component processor on the canvas that is configured for primary node only. This is configured in a processor's configuration "scheduling" tab: Primary node scheduled processors with display a "P" in the upper left corner as seen above. NOTE: ONLY processors with no inbound connections should ever be set to "primary node" execution. Doing so on processor with inbound connection can lead to FlowFiles becoming stuck in those connection when the elected primary node changes. Not all protocols are "cluster" friendly, so primary node execution helps dataflow designers work around that limitation while still benefiting from having a multi-node cluster. NiFi has numerous "List<XYZ>" and "Fetch<XYZ>" type processor typically used to handle non cluster friendly protocols. I'll use ListFile and FetchFile as an example. Let say our 3 node cluster as the sane network directory mounted to every node. If I was to add the ListFile processor and leave it configured with "all nodes" execution and configure it to list files on that shared mount. All three nodes in the NiFi cluster would produce FlowFiles for all the files listed (so you have files in triplicate). Now if i were to configure my ListFile with "primary node" execution, the listFile would only get scheduled to execute on the currently elected primary node (these processor also record cluster state in ZK so that if a elected primary node changes it doe snot result in a re-listing of the same files again). To prevent overloading the primary node, the list based processors do not retrieve the source content. It only creates a 0 byte FlowFile with attributes/metadata about the source file. So the List based processor would then be connected downstream to its corresponding FetchFile processor. The FetchFile for example would the use the metadata from the 0 byte FlowFile to fetch the content and add it to the FlowFile. On the connection between ListFile and FetchFile you would configure cluster load balancing. Here you can see I selected basic round robin. You'll notice a connection with load balancing configured will also render a bit different: What happen on this connection is that all the 0 bytes FlowFiles will be redistributed in round robin style to all connected nodes. Then on each node the FetchFile will get each nodes subset of FlowFiles content. This reduce need to transmit content of network between nodes and reduces disk IO on primary node since it is not fetching all the content. If you search the Apache NiFi documentation you will see many list and fetch combination type processors. But any source processor (one with no inbound connection) could be configured for primary node only. But only schedule a source processor as primary node execution if required. Doing so on processors like ConsumeKafka for example that uses cluster friendly protocols would just impact performance. Hope this answers your question only what the difference is between Cluster Coordinator and Primary Node roles in a NiFi cluster. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
05-03-2024
08:04 AM
1 Kudo
@manishg What reason is NiFi giving for the node disconnection? Go to NIFi global menu in upper right corner of UI of node that is still connected to cluster and selected "cluster": From the new UI you will see a list of your nodes. To the left of nodes you will see a small "view details" icon. Click on that for one of the nodes that experienced a disconnection (It might currently be connected). This will open a new UI that will contain node events. Node disconnections and reconnection with reason will be provided here. Probably the most common unexpected disconnect reason is "lack of heartbeat". Within the nifi.properties file you can configure the node heartbeat interval (nifi.cluster.protocol.heartbeat.interval) in the cluster section. The default is 5 secs and same value must be set on all nodes in your cluster. This setting control how often a node will attempt to send a heartbeat to the currently elected cluster coordinator node. The elected cluster coordinator expects to receive at least oe successful heartbeat from a node within 8 times the configured heartbeat interval. so with default 5 sec interval the cluster coordinator needs to receive a heartbeat at least once every 40 seconds or the cluster coordinator will disconnect the node. If reason is lack of heartbeat the node events will also tell you the time of that event. NiFi will auto-reconnect a node back in to the cluster if a successful heartbeat is received after a disconnection due to lack of heartbeat occurred. Things like long JVM Garbage Collection events can result in disconnects. Sometimes resolving the issue is as simple as increasing the heartbeat interval allowing more time before a node gets disconnected (for example setting to 15 secs on all nodes). Long garbage collection pauses can happen with large heap settings, large flows processing lots of FlowFiles. From the same Cluster UI, you can select the "JVM" tab to see basic details about the JVM on each node and see JVM GC details. This help you identify if GC is disproportionate amongst your nodes. CPU saturation on a node can also affect the heartbeat interval. From the same cluster UI you can select the "System" tab which will tell you the number of cores you have per node and the core load average per node. High core load average can result in longer times for any given thread to complete. Often dataflow design can contribute to core load and GC resulting in node disconnections due to lack of heartbeat. Example, one node in your cluster is executing on way more FlowFiles then any of the other nodes. This means that your design is not handling FlowFile distribution well causing one node to do much more work. Also keep in mind that the nifi-app.log will log node events as well and it may help to inspect those logs to see if any other notable logged events happened around that same time. Was the node that got disconnect the currently elected primary node (you could tell by logs in another node reporting it as being elected as primary node just after the previous elected primary node was disconnected. If that pattern is consistent, then your dataflow may heavily use "primary node" only scheduled processors and you are not handling FlowFile load balancing programmatically in your dataflow design(s). Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
05-03-2024
07:24 AM
1 Kudo
@manishg I recommend starting a new community question with your detailed query. This community question is 5.5 years old for a 5.5 year old version of NiFi. A lot has evolved and changed since then. Make sure with your query you include details about your NiFi version. Thank you, Matt
... View more
04-30-2024
05:59 AM
@Vikas-Nifi Looks like you added SSL Context Service as a custom dynamic property in this processor. SSL Context Service was added to the PutSlack processor in Apache NiFi 1.24: https://issues.apache.org/jira/browse/NIFI-12277 You can't use Dynamic properties to add controller service references in any NiFi processor. Processor code is required for this. This processor treats dynamic properties in a specific way: The processing of that dynamic property is what is causing your json parsing exception. This processor was deprecated newer versions of Apache NiFi in favor of the new PublishSlack processor. Looks like you will need to upgrade to Apache NiFi 1.24 or newer to use the SSL Context service in PutSlack or use the PublishSlack processor. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
04-29-2024
05:05 AM
@SAMSAL Without being indexed, I can't think of any other way to parse the provenance data.
... View more
04-26-2024
06:54 AM
1 Kudo
@kelpye While I am not familiar with this specific processor, maybe the following suggestions will help since the exception shared states that NiFi is unable to validate against the configured property value in the "CopybookPath" property : 1. The following string in your copybook path is a NiFi Parameter reference: #{hdfs.output.raw-input-files} When NiFi enables this processor it resolves the above the value configured to it in the parameter context list. If you replace above with the absolute local path instead of using a parameter, does it validate? 2. Also keep in mind that all components executing on the canvas are executed as the NiFi service user and not as the authenticated user who added those components to the canvas. So make sure that the NiFi service user is able to navigate the resolved path to your copybook.cbl file and has proper OS permissions to read that file. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
04-26-2024
06:40 AM
1 Kudo
@SAMSAL You can add additional attributes that you want to indexed with provenance that you could then use in your provenance searches. Take a look at the following properties available with the Write Ahead Provenance Repository: Since you want to be able to search on some FlowFile attribute, you would add it to the "nifi.provenance.repository.indexed.attributes". Keep in mind that adding additional indexed attributes or fields will increase the size of your provenance_repository disk usage. Added attributes or fields will start being indexed after restart of your NiFi. NiFi can not go back and reindex already processed FlowFiles, but this should help you going forward. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
04-26-2024
06:28 AM
@AlexisRub Not sure how to answer that for you. Typically production users who have access to a corporately managed LDAP/AD would use that with their NiFi. This provide better security as corporate can mange that adding of new users or removal of users no longer with the organization. If you also setup the ldap-user-group-provider in NiFi authorizers.xml along with setting of the ldap-provider in the login-identity-providers.xml you'll have a proper production setup. Let's say a new person joins the company and is added to the AD. the ldap-user-group-provider (depending on filters) could automatically pull in that new user identity to NiFi allowing your NiFi admin to setup access policies for them easily. And with the ldap-provider that user could then authenticate to your NiFi (successful authentication does not mean they would have authorized access). Even better is this opens the ability to use ldap/AD managed groups for authorization. Let's say you have AD group named nifiadmins. You could sync this group and its members to NiFi via the ldap-user-group-provider and set up local authorization policies using that group identity. So later some user is added or removed from the AD "nifiadmins" group. When NiFi syncs with ldap/AD via ldap -user-group-provider (default is every 30 mins), that user would be added or removed as a known member of that group and would gain or lose authorizations without needing any manual action within NiFi to make that happen. This is most common setup fro production end users with established ldap/AD groups for different teams that will access NiFi. Different teams can then be authorized access to only specific process groups and actions. I setup a local ldap which creates a bunch of fake users and groups that i can manage for testing purposes., but not something I would do in a production setup. I would leave the corporate management of user to those responsible for that access control. Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more