Member since
07-30-2019
3387
Posts
1617
Kudos Received
999
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 189 | 11-05-2025 11:01 AM | |
| 399 | 10-20-2025 06:29 AM | |
| 539 | 10-10-2025 08:03 AM | |
| 371 | 10-08-2025 10:52 AM | |
| 408 | 10-08-2025 10:36 AM |
03-14-2017
05:48 PM
4 Kudos
@Mohammed El Moumni Each Node in a NiFi cluster runs its own copy of the dataflow and works on its own set of FlowFiles. Looking at the screenshot you have above of your queue list, you can see that the two FlowFiles are not on the same node. So each node is running a MergeContent processor and each node is waiting for another FlowFile to complete their bins. You will need to look back earlier in your dataflow to see how your data is being ingested by your nodes to make sure that the matching sets of files end up on the same node for merging. Thanks, Matt
... View more
03-14-2017
02:46 PM
1 Kudo
@matthew N Any time you see the following: Unable to perform the desired action due to insufficient permissions. Contact the system administrator. You are having an authorization issue and not an authentication issue. If you look tail your nifi-app.log while you try to login again, you will see two lines output. One will state successful authentication. 2017-03-14 14:36:43,402 INFO [NiFi Web Server-5418] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<cn=Matt,ou=People,dc=sme,dc=nifi><CN=nifi-11.openstacklocal, OU=SME, O=NIFI, L=Baltimore, ST=MD, C=US><CN=nifi-13.openstacklocal, OU=SME, O=HWX, L=Baltimore, ST=MD, C=US>) GET https://nifi-11.openstacklocal:9091/nifi-api/flow/config (source ip: x.x.x.x)
2017-03-14 14:36:43,402 INFO [NiFi Web Server-5418] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for cn=Matt,ou=People,dc=sme,dc=nifi Then you will see a line that contains the above response. You need to make sure that the DN shown in this log matches exactly with what you had configured for your "Initial Admin Identity". You entry in LDAP may have uppercase CN, OU, ..., but in your nifi use log you may see lower case cn=, ou, ... It must match what is in the user.log. You can also take a look in the users.xml file that NiFi generates to make sure your user exists. It will have a UUID assigned to your user. Take that UUID and check which resources that UUID has been granted access to in the authorizations.xml file. For an ADMIN user, you will need at a minimum: /flow
/controller
/policies
/tenants Thanks, Matt
... View more
03-14-2017
02:18 PM
@Christophe Vico
Confirm the DN of the PrivateKeyEntry in the keystore of all you nodes. # keystore -v --list -keystore <keystorename> Also confirm the patterns have been set on all nodes in the nifi.properties file. All it takes is a slight difference (extra space, missing space, case sensitivity, etc...) Thanks, Matt
... View more
03-14-2017
02:14 PM
@Christophe Vico Where in the UI did you look for the controller service? Controller services that are used by processors such as HIveConnectionPool are created within process groups and not at the core level found in the hamburger menu. You can access controller services within process groups by selecting the process group and then clicking on the "gear" icon in the operate panel:
Thanks, Matt
... View more
03-13-2017
06:32 PM
1 Kudo
@matthew N Accessing the NiFi UI requires to things to be successful: 1. User Authentication: You appear to be using LDAP to handle this part. 2. User Authorization: By default NiFi uses its internal file based authorizer (Configured in authorizers.xml) If an authenticated user lacks sufficient authorization to access a NiFi resource, you will see the "Unable to perform the desired action due to insufficient permissions. Contact the system administrator." response from NiFi. In order for an authenticated user to see the NiFi UI, they must at a minimum be granted the "view the user interface" access policy. Whichever user was configured as your "Initial admin Identity" will need to access the UI and add additional users and access policies for those users. Also keep in mind that NiFi generates the users.xml and authorizations.xml files only once the first time your NiFi is started securely. If you update who your initial admin identity is later, it will not get updated if these files already exist. If this is the first time setting up a new system, simply delete the users.xml and authorizations.xml files and restart NiFi. They will then be created again based on the current configurations in the authorizers.xml. Before updating your initial admin identity in the authorizers.xml file, I suggest looking in your nifi-user.log to versify the exact string being passed to the authorizer. It must match exactly since it is case sensitive and spaces also count as valid characters. (for example: CN= is not the same as cn=) The string you see output in the nifi-user.log is what will be passed to the authorizer. Thanks, Matt
... View more
03-13-2017
03:28 PM
@matthew N Sharing the full stack trace that goes along with the above NPE will help identify where the issue is occurring.
... View more
03-13-2017
02:26 PM
1 Kudo
@John T The PriorityAttributePrioritizer controls the order in which FlowFiles on a queue are read for processing by the next processor in a dataflow. In your case that next processor is a MergeContent processor which just places the FlowFile in a Bin and move on to the next priority FlowFile. There is no definable merging order with the TAR format in the NiFi MergeContent processor. If you know the exact number of fragments that are going in to your bundle, you could try setting the fragment identifiers on the incoming FlowFiles to force a merging order. This would require you to change your merge strategy to "Defragment". Thanks,
Matt
... View more
03-13-2017
02:11 PM
1 Kudo
@Paras Mehta Unless the SQL statement in the content of each of your FlowFiles is identical, a new transaction will be created for each FlowFile rather then multiple FlowFiles in a single transaction. This is because of how the putSQL is designed to do batch inserts. The SQL command may use the ? to escape parameters. In this case, the
parameters to use must exist as FlowFile attributes with the naming
convention sql.args.N.type and sql.args.N.value, where N is a positive
integer. The sql.args.N.type is expected to be a number indicating the
JDBC Type. The content of the FlowFile is expected to be in UTF-8
format. So consider the case where you are inserting to a common DB table. In order to have every insert statement be exactly identical you will need to create attributes sql.args.N.type and sql.args.N.value for each column. sql.args.1.type ---> 12 sql.args.1.value ---> bob sql.args.2.type ---> 12 sql.args.2.value ---> smith sql.args.3.type --> 12 sql.args.3.value --> SME And so on.... You can use UpdateAttribute and maybe ExtractText processors to set these attributes on your FlowFiles. The you can use a ReplaceText processor to replace the content of your FlowFile with a common INSERT statement like below: INSERT INTO mydb("column1","column2","column3") VALUES(?,?,?) Now every FlowFile will have identical content and batch inserts will work in a single transaction. The "?" are replaced by the values from the attributes sql.args.1.value. sql.args.2.value, sql.args.3.value, and so on.... Thanks, Matt
... View more
03-13-2017
12:23 PM
@mel mendoza May I suggest that you open an Apache Jira against NiFi for this. Thanks, Matt
... View more
03-13-2017
12:05 PM
@Christophe Vico Exactly, so only <name> as you saw would have been passed to ranger for authorization. It is working correctly based upon your current configuration.
Can you access the UI of all of the other nodes without issue? Do there DNs match the same patterns on those nodes. If my above answer addressed your question, please accept that answer. Thank you, Matt
... View more