Member since
07-30-2019
3406
Posts
1622
Kudos Received
1008
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 185 | 12-17-2025 05:55 AM | |
| 246 | 12-15-2025 01:29 PM | |
| 183 | 12-15-2025 06:50 AM | |
| 277 | 12-05-2025 08:25 AM | |
| 464 | 12-03-2025 10:21 AM |
02-02-2021
08:26 AM
@peter_coppens While the HTTP spec may be case insensitive, NiFi Expression Language (EL) is case sensitive. Are we talking about an undefined number of unique header case sensitivity here? I would think you have a short defined list of possible header values here, correct? There are multiple EL functions you can use here: 1: ${http.headers.x-My-Header:isNull():ifElse('${http.headers.x-my-header}','${http.headers.x-My-Header}'):......} In above if subject "http.headers.x-My-Header" does not exist on the FlowFile it returns the value from "http.headers.x-my-header" attribute instead; otherwise it returns value from "http.headers.x-My-Header". Then you can continue the EL statement to do whatever additional manipulation needed. 2: ${allMatchingAttributes('(?i)http\.headers\.x-my-header'):join(''):....} In above it uses a case insensitive Java regular expression within the "AllMatchingAttribtes( )" EL function to return values from all FlowFile attributes that satisfy that regex and then use the "join( )" function to merge them all together. Since we expect that only one attribute will actually match, the result is just a single value. This may be best option. 3: ${http.headers.x-My-Header:replaceNull(${http.header.X-my-header:replaceNull(${http.headers.x-my-header})}):....} In above we chain multiple "replaceNull( )" functions to iterate through all case permutations till we find one that exist as an attribute on the FlowFile. Note: In any of the above you would replace that last ":...." with the rest of your EL statement fucntions needed to manipulate the subject returned as you need. Options 1 and 3 require you to know all case permutations that may exist. Option 2 may be best choice as you do not need to know all the permutations, but can be an issue if for some reason you have multiple permutations of the same attribute on the same FlowFile. Hope this helps, Matt
... View more
02-02-2021
07:12 AM
@BhaveshP I am in complete agreement with @tusharkathpal response. But you should be able to work around this issue through a configuration change in your nifi.properties file. nifi.web.proxy.host=dev.example.com:<port number> Property description: A comma separated list of allowed HTTP Host header values to consider when NiFi is running securely and will be receiving requests to a different host[:port] than it is bound to. For example, when running in a Docker container or behind a proxy (e.g. localhost:18443, proxyhost:443). By default, this value is blank meaning NiFi should only allow requests sent to the host[:port] that NiFi is bound to. Since the hostname your client is using does not match any SAN in the individual nodes certificates, the above property allows NiFi to accept this additional hostname. The other option is to create new certificates for each of your NiFi nodes where "dev.example.com' is added as an additional SAN entry. Hope this helps, Matt
... View more
02-02-2021
07:02 AM
@kk_nifi This is more likely an issue in the specific Java 11 version your are using. https://bugs.openjdk.java.net/browse/JDK-8243541 Somethings you may want to try: 1. Try a different update version of Java 11 2. add the following line to the NiFi bootstrap.conf file: java.arg.<some unused number>=-Djava.locale.providers=COMPAT Note: A restart of NiFi is needed with any config file change except logback.xml. Hope this helps, Matt
... View more
02-02-2021
06:52 AM
@Umakanth Any chance you are running a NiFi cluster (multiple NiFi nodes) or you have multiple systems all trying to consume the same data from this same SFTP server? It is possible that one host finished reading the file first and removed it before the other hosts could finsih reading the same file. SFTP is not a cluster friendly protocol and if using this processor in a NiFi Cluster, this processor should be configured to execute on "primary node" only. Otherwise all nodes in your cluster will be fighting to consume the same source files of which you can expect to see exceptions. The GetSFTP processor is also a deprecated processor in favor of the newer listSFTP and FetchSFTP set of processors. The newer processors allow you listSFTP (primary node only and produces 0 byte FlowFiles) ---> load balanced connection (balances FlowFiles across all nodes in cluster)--> FetchSFTP ( Execute on all nodes. Retrieves specific content per FlowFile). Hope this helps, Matt
... View more
02-02-2021
06:41 AM
1 Kudo
@Arash In your 4 node NiFi cluster, what value do you have set in the "nifi.remote.input.host" property in the nifi.properties file for each of the 4 nodes? It should be the FQDN for each node and not be the same value on all 4 nodes. Form the host where MiNiFi is running, can all 4 of those FQDNs be resolved and reachable over the network? If not, MiNiFI RPG is only going to be able to send successfully to one FQDN it can reach. When the RPG is started it reaches out to the URL configured in the RPG to obtain S2S details from the target host. That target host collects the host details for all currently connected nodes in the cluster and communicates that back to the client (MiNiFi). If all 4 nodes report the same configured FQDN in the "nifi.remote.input.host" property, then client only knows of one FQDN to which it can send FlowFiles over Site-To-Site (S2S). To improve redundancy in the RPG, you can provide a comma separated list of URLS in the RPG configuration so if any one node is down, the RPG can try fetch S2S details from the next host in the comma separated list. Hope this helps, Matt
... View more
02-02-2021
06:28 AM
@Abdullah If the sensitive props key value is obscured in the globals.xml file, you are running a newer version fo CFM then 1.0.0 where the bug existed where each node in the NiFi cluster ended up with a different random sensitive props key. In CFM 1.0.1 and newer, the user is required to set this property (it is not longer set to a random value when left blank). So perhaps you are having a different issue here? Did you change the sensitive props key in your CFM NiFi configs and then had an issue with starting your NiFi? I suggest starting a new question in the community since you are having a different issue than what is described in this thread.
... View more
02-01-2021
01:13 PM
1 Kudo
@JorgeO Looking at your nifi-user.log output we see the caused by line as: Caused by: org.springframework.dao.IncorrectResultSizeDataAccessException: Incorrect result size: expected 1, actual 2 This means that your LDAP/AD returned two results instead of only one. NiFi then has not idea which of those returns is the correct one it should be using. So this either an issue within your LDAP/AD or an issue within your current login-identity-providers.xml filters. But your last shared file looks fine to me. I would suggest using the ldapsearch command to run a ldap query outside of NiFi to see what returns you get for your admin user (cn=<admin user>) Hope this helps, Matt
... View more
01-26-2021
07:15 AM
1 Kudo
@JorgeO I am not clear on what "nifi screen does not show any change" means. Some things to keep in mind: 1. NiFi must first be secured before NiFi will even support any form of user authentication or authorization. http://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security_properties 2. Authentication and authorization processes are handled independently. A user must first successfully authenticate and only then will the user identity be passed to the authorization process to verify authorized policies granted for that user. 3. The ldap-provider does not sync users from ldap, it is only used to validate user credentials supplied at a login prompt with ldap to authenticate that user. Are you seeing a login window with your NiFi? (will not see this if NiFi is not yet secured). 4. Once your user successfully authenticates then authorization comes in to play. Looking at your authorizers.xml, one issue stands out to me: - You configured: <property name="Initial Admin Identity">cn=admin,dc=example,dc=local</property> This is set in the "file-access-policy-provider". This provider is used to seed the initial minimum required policy that an admin user would need in order to access the UI, add additional users to the authorizer through UI and associate those additional users to various NiFi authorization policies. - If you are a multi-node NiFi cluster, the node certificate DNs for each of yoru secured nodes also need to be configured in the "file-access-policy-provider": <property name="Node Identity 1">CN=node1, OU=nifi</property>
<property name="Node Identity 2">CN=node2, OU=nifi</property>
etc... - Problem here is that I don't see where you create that initial admin user. NiFi can not seed policies for a user that does not exist yet as a known identity to NiFi. Multiple methods are available for adding users and groups to NiFi for policy assignment. Your setup shared is using the "file-user-group-provider" which uses locally defined user/client identities. However, you did not add your admin user DN to the file-user-group-provider so that NiFi adds it to the users.xml. It needs to exist before the "file-access-policy-provider" can associate policies to that user identity. <property name="Initial User Identity 1">cn=admin,dc=example,dc=local</property> - Additionally, if you have setup a NiFi multi-node cluster, the DNs from the certificate used to secure each of the nodes must also be added as "Initial User Identity 2,3,4..." properties in the file-user-group-provider also. All authentication and authorization actions would be logged in the nifi-user.log. So check that to see what is happening if you are having access or authorization exceptions. If this log does not exist or is empty, odds are that your NiFi was not secured. In addition to the security properties I provided the link to above, you also need to make sure you set the following additional properties in the nifi.properties: nifi.web.https.host The HTTPS host. It is blank by default. nifi.web.https.port The HTTPS port. It is blank by default. When configuring NiFi to run securely, this port should be configured. The "nifi.web.https.port" is essentially the switch that enables secured NiFi when set. The default is "nifi.web.http.port" which is a non-secured port. Hope this helps, Matt
... View more
01-26-2021
06:16 AM
@SAMSAL The NiFi login-identity-providers.xml is not used to sync users from ldap. Configuring this will not result in any user being added in to the list of users within NiFi's UI. This "ldap-provider" is a login provider which when configured provides and additional user authentication method for accessing NiFi's UI. Configuring the ldap-provider does not disable TLS based authentication. It simply changes the TSL handshake from a "require" client certificate to a "want" client certificate request from NiFi. If the client does not present a client certificate during the TLS handshake, NiFi redirects to the login UI where users can enter their ldap/AD username and password to authenticate themselves to NiFi. Users and groups can be synced from ldap, but that is done within the NiFi authorizers.xml file using the ldap-user-group-provider. I also noticed from your ldap-provider login that you are using "SIMPLE" Authentication Strategy which means that none of the TLS properties are used, so no need to set them. Hope this helps, Matt
... View more
01-26-2021
06:00 AM
@louis_allen You should avoid asking additional questions on an existing post that already has an accepted solution. If the accepted solution does not resolve your issue that you may be having a different problem. You would get better visibility and a help if you were to start a new question. That being said, if you are having a "javax.net.ssl.SSLHandshakeException", you issue is during client authentication and have not even the point of verifying client authorization. So you need to take a closer look at the keystores and truststores you have setup on both your NiFi and NiFi-Registry to make sure that a mutual TLS handshake can be successfully negotiated. Feel free to ping me if you start a new community question. Hope this helps, Matt
... View more