Member since
07-30-2019
3410
Posts
1623
Kudos Received
1008
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 369 | 12-17-2025 05:55 AM | |
| 430 | 12-15-2025 01:29 PM | |
| 449 | 12-15-2025 06:50 AM | |
| 379 | 12-05-2025 08:25 AM | |
| 623 | 12-03-2025 10:21 AM |
01-24-2017
01:33 PM
2 Kudos
@Anishkumar Valsalam NiFi supports three types of user authentication strategies: 1. SSL Certificates - This is the default authentication strategy and is always on. NiFi will always accept a user supplied SSL certificate in the connection before trying any other configured authentication strategy. If a SSL certificate is not presented in the connection, NiFi will close the connection unless another authentication strategy has been configured. Those other strategy must be configured and include: 2. LDAP - NiFi can be configured to fall back on this strategy which can be configured to connect to either LDAP or LDAPS. If your user use LDAP, they do not need to have user SSL certificates issued to them. If they do have SSL certificates loaded in their browser, NiFi will accept them and will not even try using LDAP to authenticate that user. 3. Kerberos - NiFi can be configured to fall back on this strategy. If they do have SSL certificates loaded in their browser, NiFi will accept them and will not even try using Kerberos to authenticate that user. Note: Alternate authentication strategies are configured in NiFi's login-identity-providers.xml file. NiFi SSL certificate authentication is the default (first strategy that is always attempted) and cannot be disabled. NiFi will only try one of the other authentication strategies (if configured) if SSL certificates are not presented for the user. Note: Only one alternative authentication Strategy can be configured. While SSL is always enabled, you can only add either LDAP or Kerberos, but not both. Note: Authentication is only step one or a two step process that needs to be successful before a user can successfully access NiFi. NiFi authorization occurs after successful authentication. User who can successfully authenticate via any one of the above authentication strategies still need to be authorized within NiFi for access to specific NiFi resources and components. Authorization of users requires adding those users to the configured authorizer and granted those added users access to whichever access policies are needed by that user. Thanks, Matt
... View more
01-20-2017
05:23 PM
@Anishkumar Valsalam NiFi user authentication supports LDAP, Kerberos, or Users TLS certificates (default) only.
I thought you were already using LDAP for user authentication?
What do you have configured in your login-identity-providers.xml? Matt
... View more
01-20-2017
04:57 PM
@Anishkumar Valsalam The log you provided shows that user authentication is successful. What is failing is the users authorization inside NiFi. When setting up NiFi securely for the first time (enable SSL), you need to provide an "Initial Admin Identity". The Identity provided here will be used to seed the admin related access policies (Authorizations) within NiFi which will give that user the ability to access the UI. The issue you are running in to is that what your LDAP has returned and what you have configured in your authorizers.xml file do not match exactly. Configured value: EMAILADDRESS=07anis@gmail.com,CN=admin,OU=EDM,O=XXX,L=HK,ST=HK,C=HK LDAP value: EMAILADDRESS=07anis@gmail.com, CN=admin, OU=EDM, O=XXX, L=HK, ST=HK, C=HK These are case sensitive and spaces do count. You cannot simply update the authorizers.xml config file to resolve your issue. When NiFi is started securely for the first time, users.xml and authorizations.xml files are generated based on the configs in the authorizers.xml file. Subsequent restarts of NiFi will not trigger a modification or re-creation of these files if they already exist. So to correct your issue, you will need to correct your "Initial Admin Identity" configuration so it matches what is being returned by LDAP, remove the existing users.xml and authorizations.xml files and then restart so they get created with the correct value/user identity. Once you have gained access to your NiFi UI, you can add additional users and set authorizations for those users directly form the UI. Thanks, Matt
... View more
01-20-2017
04:35 PM
@Balakrishnan Ramasamy The behavior of the input dialog box is not configurable. Once you apply the password, it is encrypted and stored in the flow.xml.gz file. If any user opens the dialog box the users is simply informed that a sensitive property has been set. It is not possible to retrieve the plain text unencrypted version of the password after it has been applied.
If you can, please accept the answer if it addressed your initial question. Thanks, Matt
... View more
01-20-2017
03:05 PM
3 Kudos
@Michal R You can use an UpdateAttribute processor to change the filename. However, this would end up with every file having the same filename. Assuming each input filename is unique excpet has a .csv extension, you could do the following:
This would essentially replace the .csv extension with .avro while leaving the rest of the original filename unchanged. Thanks, Matt
... View more
01-20-2017
02:32 PM
2 Kudos
@bala krishnan When templates are added to the canvas the components are always in a stopped state. In some cases those processors or components may even be marked as invalid because they need one or more properties set. This occurs when the component contains required sensitive properties or the processor in the deployed instance is a newer version that has had additional required properties added to it. NiFi templates when created are sanitized of any sensitive properties. This is done because the encryption of these sensitive properties id tied directly to the specific sensitive props key the users creates for each NiFi installation. The PutHiveSQL processor has a dependency on the HiveConnectionPool controller service. If this controller service cannot be started then the processor will also not be able to start. Take a look at the following: https://github.com/aperepel/nifi-api-deploy You may be able to use this approach to automate deployment and starting of you template. Matt
... View more
01-19-2017
04:03 PM
@David Arllen You have an interesting scenario here. NiFi's design with this regard is with the expectation that when you build a dataflow, any node in the cluster is expected to be able to run that dataflow. This ensure true failover capability specific to NIfi node failure. In your case, i am understanding you looking for failover at a dataflow level. So GetHTTP is configured to run "on primary node" only. Lets say DC_B is your current primary node and everything is working fine. At some point the GetHTTP starts failing to connect to the source. What you want is NiFi to detect this and switch the primary node designation to another node I the cluster in hopes that the GetHTTP processor will still be successful there. Two things immediately come to mind: 1. A dataflow may have a number of processors in different dataflows all using the "on primary node" execution. If NiFi were to switch primary node because just one of those processors was failing, it would switch for all processors. 2. What if the source was truly down and none of your nodes could connect to it. NiFi would continue to switch from node to node looking to eventually hit a node that can once again connect? This would have an affect on the other primary node flows that are working. Or am i off and you are not talking about a NiFi cluster that spans multiple data centers, but rather 3 data centers all running standalone NiFi instances? If this is the case you may be able to have a process monitor the NiFi app log for getHTTP failures and use curl to stop the GetHTTP processor on DC_B and call it to start on DC_C. You however lose your centralized management of you dataflow this way. Bottom line is that this feature does not exist in NiFi currrently. Matt
... View more
01-18-2017
01:58 PM
@Anubhav Raikar I think we have a terminology issue here. :)
A NiFi FlowFile consists of two parts, the FlowFile attributes (Metadata about the FlowFile) and the FlowFile Content (In you case this would be the actual JSON). NiFi FlowFile attributes consist of key value pairs (A property name and an associated value). So if i am understanding you correctly, you have a FlowFile which has a FlowFile Attribute of property name of "CREDIT_FLAG" and a value of "C". You are looking for a way to change the FlowFile attribute Property name from "CREDIT_FLAG" to "CR_FL". You mention a criteria the "CR_FL" also exists in your flow? Are you saying you expect "CR_FL" to also exist in this same FlowFile or some other FlowFile elsewhere in your NiFi dataflow? NiFI processors interact with the FlowFile attributes and contents of a single FlowFile and do not cross reference with other processors or FlowFiles. I cannot think of any processors that would allow you to dynamically create FlowFile attribute property names. Property names do accept Expression language. I am trying to understand the value of a dynamically created FlowFile Attribute property name. What use would that attribute have later in your dataflow. since it is dynamically created, how would you reference later in some other Expression language statement? FlowFile Attributes are not part of the FlowFile content that is written out when the FlowFile is processed by a putFile or putSFTP. They can be referenced as headers and such by processors like putSQL or invokeHTTP, but how would you do that if you don't know what the FlowFile attribute property name is? Matt
... View more
01-18-2017
01:09 PM
Can you provide a little more detail on your use case? Are you trying to dynamically create new FlowFile Attributes or are you trying to update JSON tag names in the FlowFile content's JSON itself?
The updateAttribute processor works with FlowFile Attributes using the expression language. The FlowFile's content is not read by this processor.
... View more
01-17-2017
02:38 PM
2 Kudos
@Sebastian Carroll Hard to say what any custom code is doing as far as heap usage, but some existing processors can use considerable heap space. I would say that FlowFiles attributes consume the majority of the heap space in most cases. A FlowFile consists of two parts, the FlowFile Content (which lives in the NiFi content repository) and the FlowFile Attributes (This is metadata about the FlowFile and lives in heap [1] ). While generally speaking the amount of heap that FlowFile Attributes consumes is relatively small, users can build flows that have the exact opposite affect. If the user's dataflow uses processors to read large amounts of content and write it to NiFi FlowFile Attributes, heap usage will go up rapidly. If users allow large connection queues to build within the dataflow, heap usage will go up.
- Evaluate available system memory and the configured heap size for your NiF. The heap defaults for NiFi are relatively small. They are set in the bootstrap.conf and have default values of only 512 MB min and max. This is generally to small for any significant dataflow. I recommend setting both min and max values to the same value. Adjust these values according to available free memory on your system with out going to crazy. Try 4096MB and see how that performs first. Adjusting heap setting will require a nifi restart to take affect. - Evaluate your dataflow for areas where high connection queues exist. Setting backpressure through out your dataflow is one way to keep queues from growing to large. - Evaluate your flow for anywhere where you may be extracting content form your FlowFiles in to FlowFile attributes. IS it necessary or can the amount of content extracted be reduced. - Processors like mergeContent, SplitContent, SplitText, etc can use a lot of heap depending on the incoming FlowFile(s) and configuration. For example a mergeContent configured to merge 100,000 FlowFiles is going to use a lot of heap bining that many FlowFiles. A better approach is to use to mergeContent processor in a row with the first merging 10,000 and the second merging bundles of 10 again to create the 100,000 desired end result. Same goes for SplitText. If your source FlowFile results in excess of 10,000 splits, try using two SplitText processors (First splitting by every 10,000 lines and the second splitting those by every line.) With either example above you are reducing he amount of FlowFiles held in heap memory at any given time. Notes: [1] -- NiFi uses FlowFile swapping to help reduce heap usage. FlowFile attributes live in heap memory for faster processing. If a connection exceeds the configured swap threshold (default 10,000 set in nifi.properties), NiFi begins swapping out FlowFile attributes to disk. One must remember that this swapping is per connection. This swapping is not based on any heap usage but rather by object thresholds so values may need to be adjusted based on average FlowFile Attribute size.
... View more