Member since
07-30-2019
3432
Posts
1632
Kudos Received
1012
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 116 | 01-27-2026 12:46 PM | |
| 513 | 01-13-2026 11:14 AM | |
| 1137 | 01-09-2026 06:58 AM | |
| 958 | 12-17-2025 05:55 AM | |
| 469 | 12-17-2025 05:34 AM |
09-20-2022
11:08 AM
@AyanF I would hope that if you are securing a processor such as HandleHTTPRequest processor to control access to this endpoint, that you have also secured your NiFi instance/cluster as well? SSL/TLS is a very deep subject which is a bit much to discuss end-to-end here, but i will give a 10,000 foot view to help get you started. If so, then somewhere along the line a keystore and truststore were generated to secure your NiFi. Keystore files (keystore or truststore) are not something unique to NiFi nor any NiFi processor that uses them. Keystore and truststores are used to facilitate TLS handshakes and establish a secure and encrypted connection between a client and a server. A keystore used by NiFi would contain a single "PrivateKeyEntry". This privateKeyEntry used by HandleHHTPRequest processor must have an Extended Key Usage (EKU) that support at least serverAuth (for securing NiFi itself it would need both serverAuth and clientAuth EKUs). It must also contain a SubjectAlternativeName (SAN) that contains the hostname of server on which your NiFi is running. Commonly the same keystore used to secure your NiFi instance is also used in the SSLContext service to secure components used in your dataflows but not a requirement. A Truststore contains one too many "TrustedCertEntry"s. The trustedCertEntries are the public keys for self-signed certificates, intermediate, and root Certificate Authorities (CA). TLS works the same accessing a secured NiFi or secured NiFi processor as it does accessing any other https site. The server side of the connection decides if the TLS handshake should be 1-way or mutual. In a 1 way TLS connection, the secured server provides its certificate to the client. The client verify that it trusts that server certificate. The server does not require the client to identify itself via a client certificate. This is how https://www.google.com works.The Server has the option to require/need or want a client certificate. The HandleHTTPRequest processor which would be server side of the TLS connection can be configured for one of those options. In a TLS client and server exchange the certificate DN is always used to identify the server to the client and client to the server. The Client certificate presented must be trusted by the server side as well. An truststore is included with every Java distribution. The file is named "cacerts" and it contains a bunch of trustedCertEntrys for well know CAs. You can view the contents of a keystore like the cacerts file using java's keytool command: keytool -v -list -keystore <path to keystore file>/<keystore filename> Naturally if you create your own certificate it would be self signed and the cacerts file would need to have the public cert/key for your self signed private cert/key added to it. Here is a example process for creating a keystore: https://docs.oracle.com/cd/E19509-01/820-3503/ggfen/index.html There are also services (some free) you can use to generate a signed certificate. Those services will also provide you with the public certs for your truststore. https://www.tinycert.org/ But simply having a keystore and truststore on the NiFi (server) side is not enough to establish a secured connection. The client that is connecting to you NiFi handleHTTPRequest endpoint must also have a truststore. That truststore must contain the public cert/key if using a self signed server side cert or the complete trustchain for CAs that signed your server certificate. What is a trustchain? When you get a server cert signed, it may be signed by an intermediate CA or a root CA. An intermediate CA is a CA that itself has been signed by yet another CA. A root CA is signed by itself (Owner and issuer are same DistinquishedName (DN)). The complete trustschain would involve having every public cert from intermediate CA to the rootCA. Let's say your server cert is signed by intCA1 and intCA1 is signed by rootCA. RootCA is signed by RootCA. So the complete trustschain would require having the public cert/key for both intCA1 and rootCA in the truststore. Now if you choose to have your HandleHttpRequest processor to "need authentication", the client must have a certifcate the truststore configured in your SSLContextService is capable of trusting (meaning it needs to complete trustchain in that truststore for that client cert/key. Keep in mind that any client certificate that is trusted by that truststore will be able to interface with that endpoint. I understand this is a lot, but it only scratches the surface. Hopefully it is enough to get you were you need to be to secure your NiFi and NiFi components. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-20-2022
10:23 AM
@niclyx You need to be careful with increasing concurrent tasks on components. Doing so can actually decrease performance if not done carefully. All components are requesting thread from the configured thread pools. Over subscribing on one processor can adversely affect others. Did you check disk and network I/O during timeframe of decreased performance? Did you inspect NiFi app log for any warns or errors related to writing new content to content_repository? Did you monitor JVM garbage collection stats (how often and how long they took)? Was performance throughput tied to specific processors within the dataflow(s)? Where the bottleneck points all dealing with interfacing with an external service? If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-20-2022
02:57 AM
@MattWho thanks a lot. I identified connections with problematic files, disabled load-balance and terminated them according to your method via filtering by id. It's interesting that the problematic connection ID is not showed in the nifi-app.log, but in the UI logs shows
... View more
09-16-2022
11:56 AM
@noekmc I was not clear that when you accessed the NiFi Web address you were skipping the login window completely. This means that your browser provided and alternative method of client/user authentication. When you access the NiFi web address, NiFi will always negotiate a mutual TLS handshake. This is necessary because this is how NiFi nodes authenticate with one another. If no other methods of client authentication have been configured, the mutual TLS handshake "Requires" a client certificate. When other methods of authentication are configured in NiFi, the mutual TLS handshake will "WANT" a client certificate. If no client certificate is presented, then NiFi will move on to the next configured authentication method which would spnego. Spnego based authentication is enabled when the following properties have been configured in the nifi.properties file: Make sure these two properties are clear to disable spnego auth challenge to your browser. If Spnego auth challenge is not successful, NiFi moves on to next auth method such as a configured login provider like the ldap-provider you have setup. The first step is figuring out which method (TLS client certificate or Spnego) is authenticating your user. Typically a browser will prompt you when either if these methods are invoked the first time. If you ack instead of cancel, the browser will remember that choice going forward. For TLS client auth to work, your browser must have a client certificate loaded in to it that your NiFi's truststore file is capable of trusting. For Spengo to work, Spnego must be configured in your browser. Step one: - Open an incognito browser tab (it will not have any retained cookies that would auto use a certificate or spnego) and provide the NiFi UI address. Does it redirect you immediately to the login UI. If so, you now know one of these other methods are being used. - Clear the two Spnego properties if configured in the nifi.properties file. (if already blank, then we know a TLS certificate is what is being used. - Clear browser cache and cookies. Access NiFi UI address, when prompted via browser for certifcate, cancel and you should get redirected to login window. There is not configuration change that can be made in NiFi to stop a browser from doing this. However, your decision to cancel and continue to URL without providing your certifcate should be cached by your browser so it does not ask you each time afterwards. - Try a different browser. While your certificate maybe loaded in one browser, it may not be loaded in another. Same goes for Spnego, it may not be enabled in all browsers on your client. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-15-2022
12:45 PM
@rafy The ERROR shared seems unrelated to the action of emptying a connection queue. Are you trying to delete a connection or empty a connection queue? Can you share screenshots of the actions being performed? Can you collect the full stack trace produced in the nifi-app.log when you perform the action? Thank you, Matt
... View more
09-14-2022
01:55 PM
@quimic Welcome to NiFi. The PutFile processor does not produce new content, so I am not clear on "generating 2 files instead one (one file with #7000 lines the other with #3000 lines)". What this sounds like is you have two different NiFi FlowFiles (one with 7000 lines of content and another with 3000 lines of content) being passed to your PutFile processor. The PutFile processor is then just writing those to files to the configured directory path. I am guessing maybe that you wanted your MergeContent processor to produce one FlowFile with 10000 lines? If so that requires a change to the configuration within your MergeContent processor. Are you always merging 10,000 FlowFiles to 1 FlowFile? Please share some more details and the configuration of your MergeContent processor. With NiFi you need to understand that each processor operates independent of the processor before or after it. So the processor before your MergeContent is going to be moving FlowFiles in its outbound connection feeding the MergeContent at the same time MergeContent is executing. This means that not all 10,000 Source FlowFiles may be in that connection when MergeContent starts allocating those FlowFiles to a bin. If a bin meets the min merge criteria configured at completion of bin execution, it gets merged and then MergeContent executes again and gets the new FlowFiles added to that connection since last execution. The following properties control when FlowFiles allocated to bin get merged: You may also find these articles helpful when working with MergeContent processors: https://community.cloudera.com/t5/Community-Articles/Dissecting-the-NiFi-quot-connection-quot-Heap-usage-and/ta-p/248166 https://community.cloudera.com/t5/Community-Articles/How-to-address-JVM-OutOfMemory-errors-in-NiFi/ta-p/244431 If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
09-13-2022
12:50 PM
@emaxwell A few suggested additions to this excellent article: 1. The keystore password is not required to list the keystore via the keytool command. When prompted for a password, you can just hit "enter" without entering a password. The password would be needed if you want to export the actual privateCertEntry from the keystore which is not necessary in this procedure. 2. The ldap-provider configured in the login-identity-providers.xml file has two options for how to identify the authenticated ldap user: <property name="Identity Strategy">USE_DN</property> Above will use the full Distinguished Name (DN) as you mention in the article. <property name="Identity Strategy">USE_USERNAME</property> Above will use the string entered by user at the login window as the user identity. 3. Identity mapping properties in the nifi.properties file have three properties: nifi.security.identity.mapping.pattern.<unique string>=<Java regex>
nifi.security.identity.mapping.value.<unique string>=<Resulting string when java regex matches, Java Regex capture groups can be used here>
nifi.security.identity.mapping.transform.<unique string>=<NONE, LOWER, UPPER> The <unique string> can be anything but must be same exact string for all three properties. Also patterns are checked in alphanumeric order against the identity strings. First matching pattern has its value and transform applied. So you want to make sure the more complex patterns come first to avoid having wrong pattern's value and transform being applied. So a pattern like "^(.*)$" should definitely come last. The "transform" property allows you to apply an all uppercase, all lowercase, or no transform to the resulting value string. Thanks, Matt
... View more
09-12-2022
01:52 PM
@ajignacio User and group identity strings much match identically. Your ldap-user-group-provider is syncing users and groups by the identity string found in the CN AD attribute. This is why you are seeing only the CN username and CN groupname strings in the users UI within NiFi. However, when you are logging in to NiFi to authenticate you user via the ldap-provider, the resulting user identity sting is the users full AD Distinguished Name (DN). NiFi treats different strings as different users. The ldap-provider can be changed to use the user identity string typed in the username field instead of using the full DN. This is done by changing the following property: <property name="Identity Strategy">USE_DN</property> change it to : <property name="Identity Strategy">USE_USERNAME</property> Upon successful authentication the resulting user identity is evaluated against any identity mapping patterns that may be configured in the nifi.properties file. The resulting mapped value is then passed to the configured authorizer (managed-authorizer in your setup). There the authorizers is looking up that user identity string (case sensitive) against the user strings synced by your configured users group providers. If an exact match is found both the user string and the now learned group string(s) are checked against the configured NiFi policies to determine authorization. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more