Member since
07-30-2019
3406
Posts
1622
Kudos Received
1008
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 151 | 12-17-2025 05:55 AM | |
| 212 | 12-15-2025 01:29 PM | |
| 147 | 12-15-2025 06:50 AM | |
| 264 | 12-05-2025 08:25 AM | |
| 432 | 12-03-2025 10:21 AM |
02-11-2020
10:34 AM
@JatinSab Troubleshooting this here with limited information is going to prove difficult. Does the downstream processor of the connection with the "stuck" FlowFiles have... 1. Configured for all nodes or primary node Execution? If primary node only, are the queued FlowFiles on the primary node? 2. Does processor show it has active threads? (small number in upper right corner of processor) . If so, get some thread dumps from your NiFi and inspect if the thread is progressing or hung. 3. Is back pressure being applied on any of the outbound connections? 4. Is processor getting threads to do work? What is CPU load on your system and configured "Max Timer Driven Thread Count?" It may be set to low causing a thread starved situation for your dataflow. Additionally, if you try to perform a "list queue" on the connection of issue, what result do you get? If you have a support contract with Cloudera, you can open a support case and they can help you to a solution. Thanks, Matt
... View more
02-11-2020
08:27 AM
@venu413 Are these the same FlowFiles that were stuck prior to the upgrade? If you clean out your connections, do you see the issue occur later?
... View more
02-11-2020
08:20 AM
@JatinSab Apache NiFi 1.11.1 specifically has a fix https://jira.apache.org/jira/browse/NIFI-7059 which introduced a bug with load-balanced connections. This bug is addressed in 1.11.2 and is covered in jira https://jira.apache.org/jira/browse/NIFI-7117. Thanks, Matt
... View more
02-11-2020
08:04 AM
1 Kudo
@chhaya_vishwaka Please stop your NiFi nodes and delete the local NiFi "state" directory on each of them. Then restart your NiFi nodes to see if this helps. Removing the local state will clear out the retained node information which may still contain info about your nodes before they were secured. Thanks, Matt
... View more
02-10-2020
05:57 AM
@nhemamalini Within NiFi you could construct a dataflow that uses the listHDFS processor to monitor your directory tree for creation of new files. This processor retains state so it will not list same files over and over each time it checks the HDFS directory tree. An empty (no content) NiFi FlowFile will be created for each HDFS File listed which you can route to a FetchHDFS processor that would retrieve the content adding it to the FlowFile. You can then do whatever notifications, transformations, etc via the available processors in NiFi and use the publishKafika_<version> processor to send the file to your desired Kafka topic. Hope this helps, Matt
... View more
02-10-2020
05:44 AM
1 Kudo
@chhaya_vishwaka You can see from the nifi-user.log that the DN passed to NiFi was: CN=nifi-host1, OU=nifi, O=nifi, L=SG, ST=SG, C=SG Where did cert with above DN come from? Is a cert with this DN loaded on your browser? The NiFi user log goes on to tell you that the above authenticated string was not authorized for "view the user interface": [CN=nifi-host1, OU=nifi, O=nifi, L=SG, ST=SG, C=SG], groups[] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response. This certificate does not have "Admin" anywhere in its DN. Did you maybe load your NiFi node certificates in to your browser also? When your browser prompted you to select a certificate to use, did you maybe select the wrong one? I also see from above that the entire DN is being passed for authorization which tells me you do not have an identity mapping pattern setup in your nifi.properties file to trim a portion of the DN (CN value most commonly) from the full DN. If you used "Admin" as your initial admin identity, then the resulting string after authentication and post any configured identity mapping is applied must match exactly to "Admin". https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#identity-mapping-properties As an example, the following identity mapping pattern properties would result in the above DN being trimmed so that only "nifi-host1" would be passed to the authorizer. nifi.security.identity.mapping.pattern.dn=^CN=(.*?), OU=nifi, O=nifi, L=SG, ST=SG, C=SG$
nifi.security.identity.mapping.value.dn=$1
nifi.security.identity.mapping.transform.dn=NONE Note: The transform property is used to normalize the resulting mapped value to all UPPER or all LOWER case if needed. NONE results in no transform. Also keep in mind that mapping patterns will also be applied to your server certificates DNs. So if your server certificate DN matches the pattern regex also, only the server certs CN value will be passed to authorizer. So if you use identity patterns, you are likely going to need to update your "Initial User Identity <num>" and "Node Identity <num>" in your authorizers.xml file to use the CN value instead of the full DN for each of your nodes. Hope this helps, Matt
... View more
02-07-2020
08:55 AM
@JC_ROS I struggle with how your NiFi communicates with a NiFi-Registry without clientAuth in your NiFi node's PrivateKeyEntry unless your NiFi-registry has not been secured. The NiFi nodes must be authorized in your NiFi-Registry to both Read buckets and act as a proxy. Authorizing a NiFi node requires that the node authenticate itself in the TLS handshake. Unless of course you have authorized "anonymous". Thanks, Matt
... View more
02-07-2020
08:50 AM
@JC_ROS With a secured NiFi all actions are authenticated and authorized. Thus mutual TLS authentication is required for NiFi things like: 1. Node to Node communications (Older versions of NiFi had ability to set NeedClientAuth=false in the nifi.properties to make node to node comms 1-way TLS. Newer NiFi releases removed this property resulting in mutual TLS being required here. 2. NiFi Remote Process Groups (RPG) - The S2S protocol used by the RPG required mutual TLS. The NiFI instance running the RPG is acting as a client and must present a clientAuth certificate. 3. NiFi load balanced connections The above all utilize the NiFi node keystore and truststore. Hope this helps, Matt
... View more
02-07-2020
08:23 AM
@nishank_paras Let's assume your input FlowFile content looks like this: Date, IP, Description
01-30-2020, 10.0.0.1, server1
01-30-2020, 10.0.1.2, server2
01-30-2020, 10.0.3.4, server3
01-30-2020, 10.0.4.6, server4
01-30-2020, 10.0.10.2, server5 You would configure your ReplaceText processor as follows: The Search Value contains a java regular expression which will match on your entire line and uses one capture group. The Replacement Value says to replace everything matched by the Search Value Regex with only the value from capture group 1. We then configure ReplaceText to evaluate that Regex against each line. The resulting FlowFile output to the success relationship will have content that looks like this: IP
10.0.0.1
10.0.1.2
10.0.3.4
10.0.4.6
10.0.10.2 Hope this helps, Matt
... View more
02-07-2020
07:50 AM
1 Kudo
@chhaya_vishwaka "Insufficient Permissions" indicates the issue is with user/client authorization and not authentication anymore. Note: you may or may not see the "log out" option. You should tail the nifi-user.log and then try accessing the NiFi UI to get the full log output for the user action that was attempted. For example: 2020-02-07 15:32:51,925 INFO [NiFi Web Server-26362] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[nifiuser1], groups[nifiDFMeast, users] does not have permission to access the requested resource. Unable to view the user interface. Returning Forbidden response.
2020-02-07 15:32:51,927 DEBUG [NiFi Web Server-26362] o.a.n.w.a.c.AccessDeniedExceptionMapper
org.apache.nifi.authorization.AccessDeniedException: Unable to view the user interface. If you are not ever seeing the canvas, your authenticated user does not have the "view the user interface" (/flow policy) granted to your authenticated user identity string. As you can see from above nifi-user.log output, we can confirm this. What we also learn from this log output is that the user who successfully authenticated is being identified with user string "nifiuser1" and is a member of two groups "nifiDFMeast" and "users". NiFi authorization is handled by which ever authorizer you have configured in your NiFi authorixzers.xml and referenced my the following property in your nifi.properties file: nifi.security.user.authorizer=managed-authorizer By default a secured NiFi will use a file based authorizer which relies on a "users.xml" (associates locally created known users to locally created known groups) and an "authorizations.xml" (associates the UUIDs assigned to the users and groups to various authorization policies. So using example above, my users.xml file would need to contain user "nifiuser1" and groups "nifiDFMeast" and "users" (all case sensitive). Each of those strings would be assigned a UUID. In order for my user to see the NiFi canvas, one of those assigned UUIDs would need to be assigned to the /flow policy in the authorizations.xml file. NiFi on first startup only creates the users.xml and authorizations.xml file based on initial admin identity and node Identity values configured in the authorizers.xml file. If you have already started NiFi secure and later edit these identities, those changes will not be reflected in either file. Remove these two files an restart NiFi so they are created new again with your changes (this should only be done during initial setup of a secure NiFi). Once your initial admin user is able to access the UI, all additional users and authorizations policies would be added via the NiFi UI. Hope this helps, Matt
... View more