Member since
07-30-2019
3436
Posts
1632
Kudos Received
1012
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 155 | 01-27-2026 12:46 PM | |
| 570 | 01-13-2026 11:14 AM | |
| 1256 | 01-09-2026 06:58 AM | |
| 1031 | 12-17-2025 05:55 AM | |
| 502 | 12-17-2025 05:34 AM |
02-05-2021
06:25 AM
1 Kudo
@Kamil_Valko All NiFi logging is handled by the logback.xml file. By default you will find that NiFi's logback.xml contains three unique file appenders. 1. APP_FILE 2. USER_FILE 3. BOOTSTRAP_FILE. To log anything to a different file, the first thing to do is create another appender. I recommend copying the APP_FILE appender. <appender name="FTP_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>/var/log/nifi/nifi-ftp.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
<fileNamePattern>/var/log/nifi/nifi-ftp_%d{yyyy-MM-dd_HH}.%i.log</fileNamePattern>
<maxFileSize>100MB</maxFileSize>
<maxHistory>5</maxHistory>
<totalSizeCap>1GB</totalSizeCap>
</rollingPolicy>
<immediateFlush>true</immediateFlush>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender> Then you need to add a new logger for the FetchFTP processor class. You cannot control logging for specific instance of a processor. This means all logging from all instantiations of the listFTP processor on the canvas will get handled by this new logger. <logger name="org.apache.nifi.processors.standard.FetchFTP" level="ERROR" additivity="false">
<appender-ref ref="FTP_FILE"/>
</logger> You can turn off logging for a specific processor class, but I agree with @ckumar that this is not an ideal path. But for completeness of this response, setting level to "OFF" on a logger for a specific class will stop all logging from that class. (Remember that this affects logging from all occurrences of this processor used by this NiFi) Hope this helps, Matt
... View more
02-02-2021
11:57 PM
@tusharkathpal @MattWho Thanks, Guys for the help. I have tried it out the same using following command ``` ./tls-toolkit.sh standalone -n 'nifi-node1,nifi-node2,nifi-node3' -C 'CN=admin, OU=NIFI' --nifiDnPrefix 'CN=' --nifiDnSuffix ', C=IN' -K randompassword -P randompassword -S randompassword -B randompassword -o /tmp/5/certs/ssl --subjectAlternativeNames 'dev.example.com' ``` While deployment, It's giving me following error and killing the containers 2021-02-03 07:52:41,921 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead) ----------------- Update ----------------- Above issue was introduced in the latest version (1.12.x): https://issues.apache.org/jira/browse/NIFI-7730 Issue resolved by adding 'dev.example.com' as an additional SAN entry and by upgrading the NiFi version to nifi-1.13.0 (dev)
... View more
02-02-2021
08:26 AM
@peter_coppens While the HTTP spec may be case insensitive, NiFi Expression Language (EL) is case sensitive. Are we talking about an undefined number of unique header case sensitivity here? I would think you have a short defined list of possible header values here, correct? There are multiple EL functions you can use here: 1: ${http.headers.x-My-Header:isNull():ifElse('${http.headers.x-my-header}','${http.headers.x-My-Header}'):......} In above if subject "http.headers.x-My-Header" does not exist on the FlowFile it returns the value from "http.headers.x-my-header" attribute instead; otherwise it returns value from "http.headers.x-My-Header". Then you can continue the EL statement to do whatever additional manipulation needed. 2: ${allMatchingAttributes('(?i)http\.headers\.x-my-header'):join(''):....} In above it uses a case insensitive Java regular expression within the "AllMatchingAttribtes( )" EL function to return values from all FlowFile attributes that satisfy that regex and then use the "join( )" function to merge them all together. Since we expect that only one attribute will actually match, the result is just a single value. This may be best option. 3: ${http.headers.x-My-Header:replaceNull(${http.header.X-my-header:replaceNull(${http.headers.x-my-header})}):....} In above we chain multiple "replaceNull( )" functions to iterate through all case permutations till we find one that exist as an attribute on the FlowFile. Note: In any of the above you would replace that last ":...." with the rest of your EL statement fucntions needed to manipulate the subject returned as you need. Options 1 and 3 require you to know all case permutations that may exist. Option 2 may be best choice as you do not need to know all the permutations, but can be an issue if for some reason you have multiple permutations of the same attribute on the same FlowFile. Hope this helps, Matt
... View more
02-02-2021
07:02 AM
@kk_nifi This is more likely an issue in the specific Java 11 version your are using. https://bugs.openjdk.java.net/browse/JDK-8243541 Somethings you may want to try: 1. Try a different update version of Java 11 2. add the following line to the NiFi bootstrap.conf file: java.arg.<some unused number>=-Djava.locale.providers=COMPAT Note: A restart of NiFi is needed with any config file change except logback.xml. Hope this helps, Matt
... View more
02-02-2021
06:41 AM
1 Kudo
@Arash In your 4 node NiFi cluster, what value do you have set in the "nifi.remote.input.host" property in the nifi.properties file for each of the 4 nodes? It should be the FQDN for each node and not be the same value on all 4 nodes. Form the host where MiNiFi is running, can all 4 of those FQDNs be resolved and reachable over the network? If not, MiNiFI RPG is only going to be able to send successfully to one FQDN it can reach. When the RPG is started it reaches out to the URL configured in the RPG to obtain S2S details from the target host. That target host collects the host details for all currently connected nodes in the cluster and communicates that back to the client (MiNiFi). If all 4 nodes report the same configured FQDN in the "nifi.remote.input.host" property, then client only knows of one FQDN to which it can send FlowFiles over Site-To-Site (S2S). To improve redundancy in the RPG, you can provide a comma separated list of URLS in the RPG configuration so if any one node is down, the RPG can try fetch S2S details from the next host in the comma separated list. Hope this helps, Matt
... View more
02-02-2021
06:28 AM
@Abdullah If the sensitive props key value is obscured in the globals.xml file, you are running a newer version fo CFM then 1.0.0 where the bug existed where each node in the NiFi cluster ended up with a different random sensitive props key. In CFM 1.0.1 and newer, the user is required to set this property (it is not longer set to a random value when left blank). So perhaps you are having a different issue here? Did you change the sensitive props key in your CFM NiFi configs and then had an issue with starting your NiFi? I suggest starting a new question in the community since you are having a different issue than what is described in this thread.
... View more
02-01-2021
01:13 PM
1 Kudo
@JorgeO Looking at your nifi-user.log output we see the caused by line as: Caused by: org.springframework.dao.IncorrectResultSizeDataAccessException: Incorrect result size: expected 1, actual 2 This means that your LDAP/AD returned two results instead of only one. NiFi then has not idea which of those returns is the correct one it should be using. So this either an issue within your LDAP/AD or an issue within your current login-identity-providers.xml filters. But your last shared file looks fine to me. I would suggest using the ldapsearch command to run a ldap query outside of NiFi to see what returns you get for your admin user (cn=<admin user>) Hope this helps, Matt
... View more
01-26-2021
06:16 AM
@SAMSAL The NiFi login-identity-providers.xml is not used to sync users from ldap. Configuring this will not result in any user being added in to the list of users within NiFi's UI. This "ldap-provider" is a login provider which when configured provides and additional user authentication method for accessing NiFi's UI. Configuring the ldap-provider does not disable TLS based authentication. It simply changes the TSL handshake from a "require" client certificate to a "want" client certificate request from NiFi. If the client does not present a client certificate during the TLS handshake, NiFi redirects to the login UI where users can enter their ldap/AD username and password to authenticate themselves to NiFi. Users and groups can be synced from ldap, but that is done within the NiFi authorizers.xml file using the ldap-user-group-provider. I also noticed from your ldap-provider login that you are using "SIMPLE" Authentication Strategy which means that none of the TLS properties are used, so no need to set them. Hope this helps, Matt
... View more
01-26-2021
06:00 AM
@louis_allen You should avoid asking additional questions on an existing post that already has an accepted solution. If the accepted solution does not resolve your issue that you may be having a different problem. You would get better visibility and a help if you were to start a new question. That being said, if you are having a "javax.net.ssl.SSLHandshakeException", you issue is during client authentication and have not even the point of verifying client authorization. So you need to take a closer look at the keystores and truststores you have setup on both your NiFi and NiFi-Registry to make sure that a mutual TLS handshake can be successfully negotiated. Feel free to ping me if you start a new community question. Hope this helps, Matt
... View more
01-25-2021
09:29 AM
@vchhipa If you are forced to "terminate" the processor after requesting it to stop, this can mean that you have a stuck or very long running process thread. The "terminate" does not actually kill the thread, it disassociates that JVM thread from the processor and he current FlowFile that thread is associated with. The terminated thread continues to run until it completes (NiFi does request thread to quit/exit when terminating it, but success of that varies since not all process support that ability). Any terminated threads still active will be represented in the processor by a small number in parentheses (1) displayed in its upper right corner. The previously associated FlowFile is left on the inbound connection and will be handled based on queue priority when the consuming processor is started again. If the "terminated" thread should eventually complete, any output/return from that thread including logging is just sent to null. To figure out what is going on when you have a seemingly hung thread is to get a series of NiFi thread dumps: ./nifi.sh dump <dump-filename-nodenum-01.txt> Getting at least 3 dumps at an interval of 2-5 minutes apart is usually good. What you are looking for is a thread associated with your processor class (invokeHTTP in this case) where the same thread number exists in every thread dump collected. Then you will want to luck that the thread stack to see if all are identical. If thread output changes between thread dump outputs, it indicates that thread is not hung but rather just long running. If thread dump output does not change, you'll want to dig in to that output to see what it is waiting on. Hope this helps, Matt
... View more