Member since
07-30-2019
3470
Posts
1642
Kudos Received
1018
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 259 | 05-06-2026 09:16 AM | |
| 448 | 05-04-2026 05:20 AM | |
| 318 | 05-01-2026 10:15 AM | |
| 512 | 03-23-2026 05:44 AM | |
| 388 | 02-18-2026 09:59 AM |
10-03-2023
07:36 AM
1 Kudo
@edim2525 GC kicks in around 80% of heap memory usage. You could certainly enable GC debug logging to verify that GC is executing. GC can only clean-up unused memory (Memory no linger being held by a process). I see you have three NiFi nodes. Are you only having Heap memory usage issues on the one node? I see the node with growing heap usage is the elected primary node. What processors do you have running as "primary node" execution? Does your primary node have a lot more queued FlowFiles than the other nodes? If you disconnect the primary node from your cluster which will force a new primary node to be elected, does the heap then start to grow on the new elected primary node? What version of NiFi are you running? What version of Java is your NiFi running with? Have you collected heap dumps and analyzed them to see where the heap usage is being used? Do you have any custom processors added to your NiFi? Are you using any scripting based processors where you have written your own code that is being executed within NiFi? Matt
... View more
09-29-2023
06:04 AM
@VLban From what you have shared, I don't think you are having any issues with yoru NiFi communicating with your zookeeper. When NiFi is running it sends a heartbeat message to ZK so that ZK knows that node is available. ZK is used to facilitate the election of two NiFi roles: 1. Cluster coordinator - Only one node in the NiFi cluster can be elected as cluster coordinator. The cluster coordinator is responsible for replicating requests made form any node to all nodes in the cluster. This allows for NiFi to support a zero master architecture meaning that users do not need to connect to the elected cluster coordinator node in order to make changes. Users can interact with the NiFi cluster form any node. 2. Primary node - Only one node at a time can be elected to this role. The node with this assigned role will be the only node that schedules component processors configured with "primary node" only execution. Your log output shared indicates that ZK is receiving these heartbeats from at least some of the 10 nodes (maybe all of them, but we know the node from which you got this log is talking to ZK fine) allowing for cluster coordinator election to be successful. We see that "sd-sagn-rtyev:9082" was elected with the cluster coordinator role. Once nodes aware of who the elected cluster coordinator is, they will start sending cluster heartbeats to that elected cluster coordinator. The initial set of heartbeats will be used to connect the nodes to the cluster (things like making sure all nodes are running the exact sam flow.xml.gz/flow.json.gz, have matching users.xml files, and authorizations.xml files). If your NiFi is secured (running over HTTPS), then all communications between nodes are over mutualTLS encrypted connections. Based on the exception you shared, it sounds like this connection between node(s) and the elected cluster coordinator is failing. 1. Make sure that all nodes can properly resolve the cluster hostnames to reachable IP addresses. 2. Make sure that the PrivateKeyEntry in each nodes keystore configured in the nifi.properties supports EKUs clientAuth and serverAuth, have required host SAN entry(s). 3. Make sure that the truststore used on every node contains the complete trust chain for all the privateKey entries being used by all 10 nodes. A PrivateKey may be signed by a root or intermediate CA (an intermediate CA may be signed by another intermediate CA or the root CA). A complete trust chain consists of ALL trusted public certificates from signer of the Private key to the root CA. If a MutualTLS handshake can not be established, typically one side or the other will simply close the connection. Most commonly as a result of lack of proper trust. Thus would explain the Broken pipe (write failed) as the client was unable to send heartbeat connection_request to the elected cluster coordinator. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-28-2023
01:24 PM
@Frank37 Welcome to the community. NiFi's provenance tracks specific types of provenance events and not all processors produce these types of events. So typically a processor like LogMessage would not be producing any provenance events. Provenance is used to track the lifecycle of a NiFi FlowFile tracking any time it is routed, modified (content or attributes), cloned, dropped, etc. That being said, it appears your LogMessage processor is auto-terminating it's success relationship? In that case it should be generating "DROP" provenance events. I setup a simply flow to verify this works successfully: So your issue seems unrelated to the processor specifically. First thing to check are your provenance related configuration properties in the nifi.properties file. You stated oldest event is more than a month old which is not expected unless you change your default provenance settings which only retains provenance events for 7 days. If you go to the global menu in the upper right corner of the UI and select "Data Provance", do you get any provenance results? If so, the newest will be at the top. What is the dat on the newest event? If it is old and you have active running flows, this tells me provenance has not been updating for some time. Does newest provenance event align with some change to your NiFi or upgrade/migration? maybe a configuration or corruption issue: Which provenance persistence provider are you using? What is the newest file in your provenance repository storage directory? I see you are running Apache NiFi 1.19. Was this instance always running this version or did you upgrade at some point in time? If so what version was used previously? There were some bugs in older versions of NiFi that could have rendered the provenance repository corrupt if this was upgrade from some older version at one point in time. Stopping NiFi, purging all contents of provenance repository and starting up would resolve that. Maybe an authorization issue: Has your user been authorized to "view provenance" data on this component or on the process group in which this processor resides? The authorization to "query provenance" is not that same as the authorization to "view provenance" events produced by individual components. Processors inherit authorization from the parent process group unless explicit policies have been set directly on the component. Process Group in turn inherit form parent Process groups if explicit policy is not set on child process group. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-28-2023
12:54 PM
@sarithe You may also want to take a look at Process Group (PG) FlowFile Concurrency configuration options as a possible design path since there does not appear to be any dependency between task 1 and task 2 in your description. You just want to make sure that not more than 2 tasks are executing concurrently. You move your processors that handle the 2 task executions inside two different child PGs configured with "Single FlowFile per Node" Process Group FlowFile Concurrency. Within the PG you create an input port and output port. Between these two ports you handle your task dataflow. Outside this PG (parent PG level), you handle the triggering FlowFiles. The task PGs will allow 1 FlowFile at a time to enter that PG and because of the FlowFile Concurrency setting, not allow any more FlowFiles to enter this PG until that FlowFile processes out. As you can see from above example, each task PG is only processing a single FlowFile at a time. I built this example so that task 2 always takes longer, so you see that task 1 Pg is outputting more FlowFile processed the Task 2 PG while still making sure that on two tasks are ever being executed concurrently. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-27-2023
06:29 AM
@lafi_oussama Does the zipfile actually contain files or only empty directories?
... View more
09-27-2023
06:24 AM
@techNerd I don't see a question in your post. I can only assume you are talking about missing "key" policy icon on your NiFi Flow root process group? This indicates your authenticated user is not authorized to view or modify all policies. What is also interesting from your screenshot is that the user identity displayed in upper right corner is a UUID and not "CN=sys_admin, OU=NIFI" from your user certificate. So I think you have multiple issues here with your configuration. Inspect your nifi.properties, login-identity-providers.xml, and authorizations.xml files for configuration issues. Also take note that the file-user-group-provider ONLY creates the users.xml file if it does not already exist during startup. It does not modify an already existing file. The file-access-policy-provider generates the authorizations.xml (different file from authorizers.xml) ONLY if it does not already exist at startup. It will not modify an already existing file. What version of Apache NiFi is being used? Did you maybe leave remnants of the single-user-provider or single-user-authorizer configured? If so remove these two providers from your configuration. Below is more info about the "initial admin": The intent of the "Initial Admin" is to give that user just enough authority to function as a NiFi Admin (access the UI, access to view and modify tenants/user, create new users and groups identities (assumes file based authorization configured), access to assign or remove access policies to users/groups, access the NiFi controller settings and give view modify to root process group (if first start up with no pre-existing flow.xml.gz/flow.json.gz in place.). It is not meant to grant the admin to all policies, but admin has ability to add themselves to all policies. There are often clear devisions of responsibility between admins and dataflow designers/engineers. An admin not involved with creating flows would have no need to be able to build flow, access component configurations, view content, view data provenance, etc. So policies of this nature are not assigned as part of initial admin setup. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-27-2023
05:54 AM
@manishg The first thing that comes to mind is JVM heap. You may want to collect and look at garbage collection data with large files versus small files. Second would be identifying which processor(s) the largest FlowFiles spend the most time at. For this I would suggest looking at the provenance lineage for the large FlowFiles. There is a slide bar at bottom of that lineage graph that you can scroll to see progression of FlowFile through the lineage tree. What processor follow did the FlowFile spend longest at and how are they configured? Hope this helps, Matt
... View more
09-26-2023
07:19 AM
@lafi_oussama Have you tried upgrading to the latest version of Apache NiFi? You have not shared the specific ERROR you are encountering with the UnpackContent processor. You have not shared specific around what you tried using the ExecuteStreamCommand and Python scripts. Any additional details you can provide about your zip files? Thanks, Matt
... View more
09-22-2023
07:33 AM
@sid_21m Within NiFi, authentication and authorization are handled as separate processes. Upon successful authentication, NiFi has a user identity (In your case that user identity is your Azure AD username. That user identity is passed to the authorizer to make determination as to what NiFi authorization that user has been granted. At this point nothing more is known about the authenticated user other than the user identity. The Authorizer is configured in the authorizers.xml NiFi configuration file. In here you have multiple choices available to you, but none of them are capable of collecting App Roles from Azure. You can use the ldap-user-group-provider to collect ldap user to group associations from Azure AD. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more
09-21-2023
01:14 PM
@edim2525 You can't have NiFi nodes in a cluster configured for different methods of authentication and authorization. User requests made on any node are proxied to the Cluster coordinator and then replicated to all nodes. It is unlikely that your user identity will remain the same once you change to using ldap for user authentication. Then you be setting up authorization based on those new user identities. Assuming you are currently using a managed-authorizer which uses the file-user-group-provider and file-access-policy-provider in your NiFi authorizers.xml? The ldap-provider can be configured to use the LDAP/AD DN (USE_DN) or the username entered at the login prompt (USE_USERNAME) as the user identity (case sensitive). Before making any changes to authentication, you could add the the new ldap based user identity in to your NiFi and authorize that user to all policies granted your current certificate based user already has. Then make a copy of the the login-identity-providers.xml file and Edit to add the the ldap-provider. Copy modified login-identity-providers.xml to all nodes. Then modify nifi.properties file on all nodes by changing following line: nifi.security.user.login.identity.provider=ldap-provider ***Theoretically (never done this) with authorization setup for your new ldap user identity setup across all nodes, you could probably restart one node at a time understanding that the only node that redirect to the new ldap-provider based login window would be a node that has been restarted. This way wok since your new ldap user identity will get proxied to the other nodes which will have authorization in place. On restart of your NiFi cluster these modified configuration files will be read. Keep in mind that when no other methods of authentication are enabled, NiFi will "REQUIRE" a client certificate for authentication through a mutualTLS exchange. Once additional methods of user authentication is added, mutualTLS auth is always enabled and attempted first, but instead of "REQUIRE", NiFI will "WANT" a client certificate. Only when no client certificate is presented during the MutualTLS exchange will NiFi move on to next configured method of authentication (ldap in your case). MutualTLS can NOT be disabled because it is only method of authentication for node to node communications. Now a caveat to above is that I have no idea about your current configuration, current user(s), how you plan to configure your ldap-provider, if you are using LDAP or AD, etc..., so guidance is very high level here. If you found any of the suggestions/solutions provided helped you with your issue, please take a moment to login and click "Accept as Solution" on one or more of them that helped. Thank you, Matt
... View more