While I usually recommend using the existing processors to perform individual tasks and chain them together to achieve your overall goal, I think this is a case where an ExecuteScript processor with a custom script could be best. As long as the input is not on the order of 10 MB+ per flowfile, you should be able to perform text searching and counting pretty well with a simple Ruby, Groovy, or Python script and provide it in the output you want to route directly to the PutEmail processor. Otherwise, everything you want can be easily done with native processors except counting occurrences of a specific string, but you could use ExecuteStreamCommand with awk to achieve this. You'll just have to spend extra time converting the formats back and forth to be useful. ... View more

NiFi cannot be configured to use a PEM encoded certificate file ( *.crt) and key file (*.key) directly. These files must be converted into Java Keystore (*.jks) files (or PKCS12 (*.p12) keystores, but JKS is preferred). Convert the certificate from PEM to PKCS12 using openssl openssl pkcs12 -export -out keystore.p12 -in mydomain.crt Provide a strong password when prompted Create a JKS truststore file and then delete the temporary alias keytool -genkey -keyalg RSA -alias temp -keystore truststore.jks keytool -delete -alias temp -keystore truststore.jks You can populate the temporary key with any values, or leave each field empty Import the IT CA public key certificate into the truststore You don't mention receiving a separate file containing it, so it may be included in the mydomain.crt file or accessible by contacting your IT department or running other commands. See section below keytool -import -v -trustcacerts -alias domain_ca -file domainCA.pem -keystore truststore.jks When prompted to trust this certificate, type "yes" Create a JKS keystore file and then delete the temporary alias keytool -genkey -keyalg RSA -alias temp -keystore keystore.jks keytool -delete -alias temp -keystore keystore.jks You can populate the temporary key with any values, or leave each field empty Import the private key into the keystore keytool -v -importkeystore -srckeystore keystore.p12 -srcstoretype PKCS12 -destkeystore keystore.jks -deststoretype JKS You now have a keystore.jks and truststore.jks file to use with NiFi. In nifi.properties, provide the paths to these files in nifi.security.keystore and nifi.security.truststore respectively, provide "JKS" as the value for nifi.security.keystoreType and nifi.security.truststoreType , and the respective password for each in nifi.security.keystorePasswd and nifi.security.truststorePasswd . Obtaining the CA certificate There are multiple ways to obtain the CA certificate containing the public key. The mydomain.crt file contains a complete certificate chain You can determine if this is the case by viewing the encoded contents of the file using a text editor, more, cat, etc. If there are multiple -----BEGIN CERTIFICATE----- lines in the file, there are multiple certificates stored in it If this is the case, you can use the command below to view the chain in parsed form, and find the certificate whose "Subject: " looks like C=US, ST=CA, L=Santa Monica, O=Unknown, OU=Your IT Organization, CN=mydomain.com CA. The order in the output matches the order of the blocks in the mydomain.crt file, but usually it will be the last block openssl crl2pkcs7 -nocrl -certfile mydomain.crt | openssl pkcs7 -print_certs -text -noout Copy the encoded block representing the CA certificate into a new file called domainCA.pem. From here, return to Step 3.2 above You can request the CA certificate from your IT department If this is the case, they will likely provide it in PEM format, with either a domainCA.pem or domainCA.crt extension. From here, return to Step 3.2 above You can obtain the CA certificate by connecting to the CA hostname and downloading the presented certificate Determine the hostname of the CA. If it is not available via the network because it is kept offline (the most secure practice), you'll have to connect to an intermediate CA (this can be determined from the certificate chain) For example, we will pretend mydomain.crt was signed by Let's Encrypt, a free commercial certificate authority. This will take the place of your (similar) ca.domain.com CA Use the following command to connect to the CA and view its certificates. You'll copy and paste the output that shows a PEM certificate to a file called domainCA.pem echo 'Q' | openssl s_client -connect letsencrypt.org:443 -showcerts In the output, you'll see a section titled "Certificate chain" followed by a zero-indexed list of certificates. Each consists of three pieces: The first line is the index and subject of the certificate. This is "who the certificate is" 0 s:/CN=www.letsencrypt.org The second line is the issuer of the certificate. This is "who says this certificate is who it claims" i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 The remaining lines are the PEM encoded certificate block. When you reach the last certificate in the list, this is the section to copy to domainCA.pem -----BEGIN CERTIFICATE----- MIIH5jCCBs6gAwIBAgISA2gSCm/BtvCR2e2bIap5YbXaMA0GCSqGSIb3DQEBCwUA ... l1Ou20Dm9TxnNw== -----END CERTIFICATE----- ... View more

Please specify which service this error is appearing on and provide full stacktraces from the respective logs to help diagnose the issue. You can also use the openssl s_client command to verify the certificates presented by each service. ... View more

You should ensure that both the NiFi instance and NiFi Registry instance have certificates which include the hostname (in DNS form) in the Subject Alternative Name section to allow RFC 6125 compliant hostname verification. There is a video tutorial for this process available here. ... View more

Justin, A couple other things to look at: If your certificate is invalid for any reason (expired, not yet valid, revoked, etc.) it can manifest in a cipher suite mismatch or no cipher suites in common error because any cipher suite that relies on RSA signatures or encryption will be unavailable. Double check that the certificate validity dates are correct and the hostname and SAN match. You can use some open source tools to check the list of cipher suites your server supports -- I recommend cipherscan as that is what I usually use, but TLSSLed is another one I have heard of. I am not as familiar with Windows as Mac OS X/*nix systems, but you can have OpenSSL list the cipher suites available to it (as a client) as well with openssl ciphers -V -tls1 (NiFi 1.2+ only supports incoming TLSv1.2 connections, but there isn't a flag for TLSv1.2 only in OpenSSL yet). If you prefer to do the scanning manually, you can use the script below from https://www.securityevaluators.com/using-openssl-determine-ciphers-enabled-server/. for v in ssl2 ssl3 tls1 tls1_1 tls1_2; do for c in $(openssl ciphers 'ALL:eNULL' | tr ':' ' '); do openssl s_client -connect <server>:<port> \ -cipher $c -$v < /dev/null > /dev/null 2>&1 && echo -e "$v:\t$c" done done ... View more

Justin, I made a diagram to hopefully make this clearer. The desired scenario you are describing -- NiFi secured with a certificate signed by an external, trusted certificate authority -- is definitely supported. You just have to ensure that the correct entities have the correct public and private keys, and that the trust chain is intact. In this case, you need to have the NiFi public certificate (green) be signed by the Intermediate CA (orange), the client public certificate (blue) signed by the Intermediate CA (orange), and the Intermediate CA public certificate (orange) in the NiFi truststore. That tells NiFi to accept any client certificate that is signed by the Intermediate CA as valid (authorization of the DN to follow after this step). For your browsers to mark the NiFi instance as trusted, the Intermediate CA public certificate (orange) must be imported into the browser's truststore (or the client machine OS keychain, depending on the browser). This can be confusing. In general, if something possess the private key X, it is X. If it possess the public certificate X, it trusts X. The client browser needs to possess the Intermediate CA public certificate to trust that NiFi is who it claims it is. The client browser needs to possess the client private key to tell NiFi it is User Jane Han. The matching client public certificate (what the browser actually sends across the network) needs to be signed by Intermediate CA private key for NiFi to trust that Mallory Gonzalez didn't just make her own certificate that says "I am Jane Han". Similarly, there could be a different Jane Han in a different organization that is not authorized to access NiFi; her (valid) certificate that says "I am Jane Han" isn't signed by your Intermediate CA, so NiFi will reject it. The NiFi truststore can contain many public certificates. Every public certificate contained there represents a trusted signer (via the corresponding private key). In the scenario below, NiFi will trust any certificate signed by either Intermediate CA or Root CA. If you added "Bernadette's CA" public certificate to that truststore, any certificates signed by that private key would also be accepted by NiFi. The commands you copied are not the complete series of steps necessary. For one thing, I don't see any reference to truststore.jks at all. I believe you may have also generated the private key under one alias ("mydomain") in the keystore.jks file but then imported the signed public certificate via the certificate signing request to a different alias ("root"). The alias of the signed public certificate and the private key need to be the same. See this Oracle doc for those steps (note the CA public cert is imported under a different alias, but the signed public cert is imported with the original alias). ... View more

Hi Justin, Let's assign some identifiers so we're talking about the same components: Root CA (your company's root certificate authority) -- Root CA Issuing Authority (your company's intermediate CA?) -- Intermediate CA Public certificate identifying the NiFi service signed by Root CA (and/or Intermediate CA) -- nifi.pem NiFi Server Keystore containing externally-signed certificate (nifi.pem) & private key -- nifi.jks Truststore containing externally-signed certificate (nifi.pem) (and/or certificate of Root CA) -- nifi_trust.jks Client Keystore (or PEM files) containing public certificate & private key of client -- client.p12 Steps to identify the issue: If you can copy the exact error message you get from NiFi's UI when browsing with the client certificate, that will be helpful. The server truststore (nifi_trust.jks) must contain the exact certificate (or one in the chain that signed it) presented by the client. If your client certificates are still signed by the old, self-signed NiFi certificate, and NiFi's truststore no longer contains that certificate, the client certs will be rejected. NiFi does not need to use self-signed certificates at all. The TLS Toolkit is provided as a convenience tool for users who do not have a dedicated security/IT team or feel comfortable generating their own CA and certificates manually, and the toolkit does generate self-signed certs. However, any version of NiFi can use externally-signed certificates, and in current Apache NiFi master, the toolkit can even accept externally-signed certificates in place of its own self-signed CA certificate to allow for chained trust. A SAN must be present in the server certificate for modern browsers to verify the identity of the server (see RFC 6125). NiFi does enforce this when possible. You can verify that the TLS handshake negotiation works by following these steps: Convert the client keystore to individual PEM-encoded files (if not already): Export the public certificate from the PKCS12 keystore: openssl pkcs12 -in client.p12 -out client.der -nodes -password "pass:<client keystore password>" Export the private certificate from the PKCS12 keystore: openssl pkcs12 -in client.p12 -nodes -nocerts -out client.key -password "pass:<client keystore password>" If the client.der file contains "-----BEGIN PRIVATE KEY-----", remove the private key block, and rename the file client.pem If the client.der file does not contain that string, re-encode it as PEM: openssl x509 -inform der -in client.der -out client.pem Use the openssl s_client command to attempt a connection using the certificates and key: $ openssl s_client -connect <host:port> -debug -state -cert client.pem -key client.key -CAfile <path_to_your_CA_cert.pem> Please report back with the output of those steps and any additional information that may arise from what I've described. If I misunderstood your scenario or you've already tried this, please let me know. ... View more

Zack, glad you found a solution. I was coming to post this but wanted to verify something in my custom configuration wasn't affecting this before posting. We can probably improve the processor/controller service documentation to make this clearer and more intuitive. If you do decide you want client authentication and configure the truststore-related properties, you can also further restrict the authorized users via the Authorized DN Pattern processor property. ... View more

Amarnath, You are trying to resolve the issue of the untrusted server certificate if curl is run without the -k/--insecure option. That was not Zack's problem. To accept input from a client which is not identifying itself with a client certificate, the solution is to provide a StandardRestrictedSSLContextService with only the keystore-related options configured. Configuring the truststore-related options instructs the service to require a client certificate to negotiate a TLS handshake and allow the connection. This is what Zack was trying to avoid. ... View more

I have provided a feature implementation (NIFI-5476 | PR 2935) in the current Apache NiFi master branch which allows the TLS Toolkit running in standalone mode to use an externally-signed CA certificate to sign node certificates. With this capability, you can purchase/obtain a certificate signed by a known (browser-installed) CA, which will be trusted by browsers. You can then use this certificate to sign node certificates. I am currently working on enabling the same feature in client/server mode, but it is not yet present. ... View more