Member since
07-30-2019
3470
Posts
1642
Kudos Received
1018
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 301 | 05-06-2026 09:16 AM | |
| 489 | 05-04-2026 05:20 AM | |
| 355 | 05-01-2026 10:15 AM | |
| 522 | 03-23-2026 05:44 AM | |
| 394 | 02-18-2026 09:59 AM |
11-22-2021
05:12 AM
@Ankit13 I would still use Cron scheduling on the PutFile processor, but rather than just having it run once at say hour 7, I'd schedule it to run every second starting at hour 7. That may it starts putting files at hour 7 and continues to put files all the way until 07:59:59. Then it stops executing until the next day. http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html Hope this helps, Matt
... View more
11-18-2021
06:45 AM
@prova Based on timestamp shared, the source is RFC3164 syslog messages in which the timestamp does not include a year. The SyslogReader supports both RFC3164 and RFC5424 syslog messages, but uses a generic syslog schema applied against the source data: {
"type" : "record",
"name" : "nifiRecord",
"namespace" : "org.apache.nifi",
"fields" : [ {
"name" : "priority",
"type" : [ "null", "string" ]
}, {
"name" : "severity",
"type" : [ "null", "string" ]
}, {
"name" : "facility",
"type" : [ "null", "string" ]
}, {
"name" : "version",
"type" : [ "null", "string" ]
}, {
"name" : "timestamp",
"type" : [ "null", "string" ]
}, {
"name" : "hostname",
"type" : [ "null", "string" ]
}, {
"name" : "body",
"type" : [ "null", "string" ]
} ]
} You can see that timestamp is treated as a string. When it comes to reformatting the customer is looking for, where is NiFi expected to extract the year from since int is not in the syslog message? Since schema treats the timestamp as a string, it can't be treated like a timestamp type within the syslog for reformatting.This is possible with RFC5424 formatted source syslog messages. This is not to say that you could not manipulate this date string via some downstream processor, but would still need to figure out where you are going to get the year from. NiFi can't assume that RFC3164 formatted syslog message was produced in same year that NiFi is parsing it. This becomes hard to handle evening via some downstream processor at end of year where NiFi servers may already be in 2022 for example but received RFC3164 syslog messages were produced in 2021. RFC3164 was absolute when RFC5424 was introduced. RFC3164 syslog messages are produced by older systems and the options here are limited. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-16-2021
06:25 AM
@sandip87 This statement is not clear to me: We had combined them to form one certificate and then follow steps in the NiFi documentation. Sounds to me like your trusts chain has an intermediate and root CAs in it. That means your truststore must have two trustedCert Entries in it. One for intermediate CA and other for the Root CA. It sounds like you only have the root CA in your truststore. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-16-2021
06:13 AM
@Yemre The ability to dynamically fetch secrets/passwords form an external source is not something that exists currently. Doing so would require modification with the every component class that uses sensitive properties. There is some progress in this path however: https://issues.apache.org/jira/browse/NIFI-5481 This new feature handles pulling secrets from an external vault, but is a NiFi core level feature and does not extend in to individual flow component level. I recommend raising an Apache NiFi Jira with your specific request. https://issues.apache.org/jira/projects/NIFI/ If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-16-2021
05:56 AM
@emmanuel Can you share the verbose output for your NiFi keystore: keytool -v -list -keystore <nifi keystore> Does that output align with the certificate requirements for NiFi? https://docs.cloudera.com/cfm/2.1.2/cfm-security/topics/cfm-security-tls-certificate-requirements-recommendations.html What version of NiFi are your running? What version of Java is your NiFi using? If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-08-2021
06:04 AM
@sandip87 The following properties within the nifi.properties file will tell you where your NiFi's keystore and truststore files are located: 1. nifi.security.keystore 2. nifi.security.truststore You can use the java keytool command to see the verbose details of the content of these two keystores: <path to java JDK>/bin/keytool -v -list -keystore <keystore/truststore> Once you inspect the these to make sure the contents are good, then you need to make sure you can successfully authenticate your user in to your NiFi. By default once NIFi is secured, the only method to authenticate a user/client is via a mutual TLS handshake which means your user needs to have a certificate loaded in the browser. Optionally you can add additional user authentication methods if you want. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication
... View more
11-02-2021
09:38 AM
@sandip87 Without the detailed output of your NiFi keystore and truststore and the of your client certificate you are using to authenticate yourself to NiFi, it would be difficult to say exactly where your issue is. I am leaning towards and issue with your company issued certificates because you stated the this same NiFi worked fine when using certificates and keystores generated by NiFi's TLS toolkit. For NiFi's keystore (configured in nifi.properties file), make sure the following are correct: 1. keystore contains 1 and only 1 PrivateKeyEntry 2. Make sure the PrivateKeyEntry ExtendedKeyUsage (EKU) contains clientAuth and serverAuth 3. Make sure that the PrivateKeyEntry contains a SAN entry that matches the hostname of the host where NiFi is running. For the user certificate loaded in the browser being used to authenticate with this NiFi: 1. verify certificate issuer. Is it an intermediate CA or the root CA? 2. Verify the NiFi's truststore.jks (configured in nifi.properties file) contains aTrustedCertEntry for the complete trust chain that goes with your certificate and the certificate found in the keystore.jks. A complete trust chain means that the truststore has the public keys for the issuer of each of the above certificates and if that issuer is and intermediate CA, you also have the public certificate for the CA that signed that intermediate CA in the truststore. You'll know when you have reached the root CA when the TrustedCertEntry has the same DN for both owner and issuer. Your browser must also contain the complete trusts chain for the certificates issued to your NiFi nodes. Once all the above is verified, clear your browser cache and site cookies. If you still have same issue and you are using Chrome browser, try typing "thisisunsafe" (which tells chrome to skip certificate verification on the certificate presented from the NiFi instance) while the NiFi chrome tab is in focus. If this works and allows you to proceed, this again points at a trust issue between your corporately issued certificate and your browser. Go back and verify structure/content of the NiFi keystore again. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-02-2021
08:23 AM
@AnnaBea Let me make sure I am clear on your ask here: 1. You have successfully split your source file in to 3 parts (header line, body line(s), and footer line). 2. You have successfully modified all three split files as needed. 3. You are having issues re-assembling the three split files back in to one file in order of header, body, footer using MergeRecord processor? With this particular dataflow design, the MergeRecord processor is not likely what you want to use. You probably want to be using the MergeContent processor instead with a "Merge Strategy" of "Defragment". But to get these three source FlowFiles merged in a specific order would require some additional work in your upstream flow. In order to use "Defragment" your three source FlowFiles all would need o have these FlowFile Attributes: fragment.identifier All split FlowFiles produced from the same parent FlowFile will have the same randomly generated UUID added for this attribute fragment.index A one-up number that indicates the ordering of the split FlowFiles that were created from a single parent FlowFile fragment.count The number of split FlowFiles generated from the parent FlowFile 1. Add one UpdateAttribute processor before your RouteText and configure it to create the "fragement.identifier" attribute with a value of "${UUID()}" and another Attribute "Fragment.count" with a value of "3". Each FlowFIle produced by RouteText should then have these two attribute set on it. 2. Then add one UpdateAttribute processor to each of teh 3 flow paths to set the "fragment.index" attribute uniquely per each dataflow path. value=1 for header, value=2 for body, and value=3 for footer. 3. Now the MergeContent will have what it needs to bin these three files by the UUID and merge them in the proper order. There are often times many ways to solve the same use case using NiFi components. Some design choices are better than others and use less resources to accomplish the end goal. While above is one solution, there are others I am sure. Cloudera's professional services is a great resource that can help with use case designs. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-02-2021
06:54 AM
@Yemre The following response you see in the NiFi UI after supplying a username and password in the tells you that the issue happened during the user authentication process: "Unable to validate the supplied credentials. Please contact the system administrator." NiFi has not even tried to do any authorization yet, so your authorizers.xml setup has not come in to the equation yet. Unfortunately, the error produced by the openldap client is rather generic and could mean any of the following could be the issue: 1. incorrect ldap/AD manager DN 2. Incorrect ldap/AD manager password 3. Incorrect username 4. Incorrect user password 5. Incorrect user search filter in the login-identity-providers.xml file In your case it looks like number 5 may be your issue: The ldap-provider expects that the username typed in the login window is passed via the "User Search Filter" so that the entered user's credentials can be verified. I noticed you are using full DNs to login with which is extremely rare. The more common approach here is to configure your ldap-provider with "Identity strategy" of "USE_USERNAME" instead of "USE_DN". This means upon successful user authentication, it is the user string entered in the login window that is used to authorize your user instead of the user's full DN. This means your initial admin string should match your username as you would type it in at the login prompt. In order to pass the entered string at the login prompt to the ldap-provider, your "User Search Filter" would need to look something like this: <property name="User Search Filter">(cn={0})</property>
or
<property name="User Search Filter">(sAMAccountName={0})</property> You should inspect your user ldap/AD entry to see which attribute in your ldap entry contain your username that you type in the login prompt. The user entered username at login is substituted in place of "{0}" in the User Search Filter. When you change the initial admin user string from the full DN to just the username, you would need to remove the old authorizations.xml (NOT the authoirizers.xml) file that was built originally with the full DN by the file-access-policy-provider in your authorizers.xml. The authorizatiions.xml file is only seeded via the file-access-policy-provider if the file does not already exist. Once it exist all future edits to content of this file is handled via changes made from within the NiFi UI. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-02-2021
05:48 AM
@Ankit13 My recommendation would be to only automate the enabling/disabling and starting/stopping of the NiFi processor component that is ingesting the data in to your NiFi dataflow and leave all downstream processors always running, so that any data that is ingested to your dataflow has every opportunity to be processed through your dataflow to the end. When a "running" processor is schedule to execute, but has no FlowFiles queued in its inbound connection(s), it is pauses instead of running immediately over and over again to prevent excessive CPU usage, so it is safe to leave these downstream components running all the time. Thank you, Matt
... View more