Member since
07-30-2019
2906
Posts
1438
Kudos Received
844
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
45 | 04-17-2024 11:30 AM | |
51 | 04-16-2024 05:36 AM | |
32 | 04-15-2024 05:31 AM | |
118 | 04-03-2024 05:59 AM | |
129 | 04-02-2024 01:22 PM |
11-22-2021
08:54 AM
@Yemre If you have a support contract with Cloudera, I'd recommend opening a support case to assist with your issue here. Possible causes: 1. Unsuccessful Mutual TLS handshake with NiFi-Registry from the NiFi hosts resulting in NiFi node connection only being 1-way TLS connection and node treated as an "anonymous" user. Anonymous users would could not proxy user requests and can not see anything except public buckets. --- Caused by missing complete trust chain on one or both sides of connection. Truststore in NiFi-Registry contains complete trust chain for NiFi hosts keystore PrivateKeyEntry. --- Caused by PrivateKeyEntry not meeting minimum requirements (missing SAN with NiFi hostname, missing EKU of clientAuth, and/or using wildcards are the most common) 2. NiFi-registry is configured with an identity mapping pattern in the nifi-registry.properties file that is matching on the DN from the the NiFi's client certificate presented in the mutual TLS handshake. The Identity mapping value and transform is then being applied which alters the actual client string which must be then authorized for Proxy and buckets policies. Hope this helps you, Matt
... View more
11-22-2021
05:12 AM
@Ankit13 I would still use Cron scheduling on the PutFile processor, but rather than just having it run once at say hour 7, I'd schedule it to run every second starting at hour 7. That may it starts putting files at hour 7 and continues to put files all the way until 07:59:59. Then it stops executing until the next day. http://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html Hope this helps, Matt
... View more
11-18-2021
06:45 AM
@prova Based on timestamp shared, the source is RFC3164 syslog messages in which the timestamp does not include a year. The SyslogReader supports both RFC3164 and RFC5424 syslog messages, but uses a generic syslog schema applied against the source data: {
"type" : "record",
"name" : "nifiRecord",
"namespace" : "org.apache.nifi",
"fields" : [ {
"name" : "priority",
"type" : [ "null", "string" ]
}, {
"name" : "severity",
"type" : [ "null", "string" ]
}, {
"name" : "facility",
"type" : [ "null", "string" ]
}, {
"name" : "version",
"type" : [ "null", "string" ]
}, {
"name" : "timestamp",
"type" : [ "null", "string" ]
}, {
"name" : "hostname",
"type" : [ "null", "string" ]
}, {
"name" : "body",
"type" : [ "null", "string" ]
} ]
} You can see that timestamp is treated as a string. When it comes to reformatting the customer is looking for, where is NiFi expected to extract the year from since int is not in the syslog message? Since schema treats the timestamp as a string, it can't be treated like a timestamp type within the syslog for reformatting.This is possible with RFC5424 formatted source syslog messages. This is not to say that you could not manipulate this date string via some downstream processor, but would still need to figure out where you are going to get the year from. NiFi can't assume that RFC3164 formatted syslog message was produced in same year that NiFi is parsing it. This becomes hard to handle evening via some downstream processor at end of year where NiFi servers may already be in 2022 for example but received RFC3164 syslog messages were produced in 2021. RFC3164 was absolute when RFC5424 was introduced. RFC3164 syslog messages are produced by older systems and the options here are limited. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-16-2021
06:25 AM
@sandip87 This statement is not clear to me: We had combined them to form one certificate and then follow steps in the NiFi documentation. Sounds to me like your trusts chain has an intermediate and root CAs in it. That means your truststore must have two trustedCert Entries in it. One for intermediate CA and other for the Root CA. It sounds like you only have the root CA in your truststore. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-16-2021
06:20 AM
@Yemre Authorizing your user is not enough. The NiFi nodes themselves need to be able to successfully authenticate via a mutual TLS handshake with the target NiFi-Registry. Those nodes then need to be authorized to read all buckets and given read/write to proxy user requests. When a User authenticates in to NiFi, that user entity is authorized to perfrom actions based on authorizations in NiFi. When it comes to NiFi then talking to NiFi-Registry, The NiFi node is proxying request to the NiFi-Registry on behalf of the user authenticated into NiFi. Also background threads in NiFi just like the NiFi processors added to the canvas are not executing as the user authenticated in to NiFi. So in the background NiFi connects to NiFi-Registry to check on current version controlled process groups to see of newer versions exist. While you are granting your NiFi nodes the ability to read all buckets, the NiFi users should be given read and write authorizations to the specific buckets that that user is going to sue to version control their Process Group. @Yemre The ability to dynamically fetch secrets/passwords form an external source is not something that exists currently. Doing so would require modification with the every component class that uses sensitive properties. There is some progress in this path however: https://issues.apache.org/jira/browse/NIFI-5481 This new feature handles pulling secrets from an external vault, but is a NiFi core level feature and does not extend in to individual flow component level. I recommend raising an Apache NiFi Jira with your specific request. https://issues.apache.org/jira/projects/NIFI/ If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-16-2021
06:13 AM
@Yemre The ability to dynamically fetch secrets/passwords form an external source is not something that exists currently. Doing so would require modification with the every component class that uses sensitive properties. There is some progress in this path however: https://issues.apache.org/jira/browse/NIFI-5481 This new feature handles pulling secrets from an external vault, but is a NiFi core level feature and does not extend in to individual flow component level. I recommend raising an Apache NiFi Jira with your specific request. https://issues.apache.org/jira/projects/NIFI/ If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-16-2021
05:56 AM
@emmanuel Can you share the verbose output for your NiFi keystore: keytool -v -list -keystore <nifi keystore> Does that output align with the certificate requirements for NiFi? https://docs.cloudera.com/cfm/2.1.2/cfm-security/topics/cfm-security-tls-certificate-requirements-recommendations.html What version of NiFi are your running? What version of Java is your NiFi using? If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more
11-08-2021
06:04 AM
@sandip87 The following properties within the nifi.properties file will tell you where your NiFi's keystore and truststore files are located: 1. nifi.security.keystore 2. nifi.security.truststore You can use the java keytool command to see the verbose details of the content of these two keystores: <path to java JDK>/bin/keytool -v -list -keystore <keystore/truststore> Once you inspect the these to make sure the contents are good, then you need to make sure you can successfully authenticate your user in to your NiFi. By default once NIFi is secured, the only method to authenticate a user/client is via a mutual TLS handshake which means your user needs to have a certificate loaded in the browser. Optionally you can add additional user authentication methods if you want. https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication
... View more
11-02-2021
09:55 AM
@Sridhara47 A status code 302 means there was a redirect in the response. I'd suggest inspecting the NiFi app.log for this exception and take a look at the full stack trace if it exists to see what the issue may be. Hope this helps, Matt
... View more
11-02-2021
09:38 AM
@sandip87 Without the detailed output of your NiFi keystore and truststore and the of your client certificate you are using to authenticate yourself to NiFi, it would be difficult to say exactly where your issue is. I am leaning towards and issue with your company issued certificates because you stated the this same NiFi worked fine when using certificates and keystores generated by NiFi's TLS toolkit. For NiFi's keystore (configured in nifi.properties file), make sure the following are correct: 1. keystore contains 1 and only 1 PrivateKeyEntry 2. Make sure the PrivateKeyEntry ExtendedKeyUsage (EKU) contains clientAuth and serverAuth 3. Make sure that the PrivateKeyEntry contains a SAN entry that matches the hostname of the host where NiFi is running. For the user certificate loaded in the browser being used to authenticate with this NiFi: 1. verify certificate issuer. Is it an intermediate CA or the root CA? 2. Verify the NiFi's truststore.jks (configured in nifi.properties file) contains aTrustedCertEntry for the complete trust chain that goes with your certificate and the certificate found in the keystore.jks. A complete trust chain means that the truststore has the public keys for the issuer of each of the above certificates and if that issuer is and intermediate CA, you also have the public certificate for the CA that signed that intermediate CA in the truststore. You'll know when you have reached the root CA when the TrustedCertEntry has the same DN for both owner and issuer. Your browser must also contain the complete trusts chain for the certificates issued to your NiFi nodes. Once all the above is verified, clear your browser cache and site cookies. If you still have same issue and you are using Chrome browser, try typing "thisisunsafe" (which tells chrome to skip certificate verification on the certificate presented from the NiFi instance) while the NiFi chrome tab is in focus. If this works and allows you to proceed, this again points at a trust issue between your corporately issued certificate and your browser. Go back and verify structure/content of the NiFi keystore again. If you found this response assisted with your query, please take a moment to login and click on "Accept as Solution" below this post. Thank you, Matt
... View more