Securing Nifi with SSL and using OIDC provider for users authentication

[#1]

Hi @MattWho 

Need your help and suggestion here as i went through many of the Nifi related article here , since my use case is different need your valuable suggestion .

1. i want to know how to use SSL\TLS to have https mode for nifi url 
2. in our organization already we are generating signed certificate with SAN entries as
nifi-dev.x.x.net [domain we try to access via browser]
nifinode1.x.x.net
nifinode2.x.x.net
nifinode3.x.x.net
zookeepernode1.x.x.net
zookeepernode2.x.x.net
zookeepernode3.x.x.net
post request raised we get a file nifi.crt file with this what else certificates are required\Needed?
3. as trial basis tried already creating keystore and truststore but seems some issue 
4. 

5. 

[#2 · Accepted solution]

@pavanshettyg5 
The TLS implementation for NiFi requires proper configuration of both keystore and truststore your organization's signed certificate with SAN entries to enable secure HTTPS access. Based on your description, you've received a signed certificate (nifi.crt) but are experiencing issues with the complete TLS setup.
 Required Certificates & Files

• Certificate Authority (CA) Certificate: The root certificate from your signing authority
• Truststore: Contains the Certificate Authority (CA) root/intermediate certificates that signed your NiFi certificate (for mutual TLS or cluster communication).
• Signed Certificate (nifi.crt): Your domain certificate with the SAN entries
• Private Key: The private key used to generate the CSR

1. Prepare Certificate Files

Ensure you have:

• The signed certificate (nifi.crt)
• Your private key
• The CA certificate (request from your CA if not available)

Problem 1: Missing Private Key or Certificate Chain

• If you only have nifi.crt, you must also have:

  • The private key (e.g. nifi.key) generated during the CSR process.

  • The CA root/intermediate certificates (if your organization uses a private CA).

Problem 2: Improper Keystore/Truststore Format

• NiFi uses Java KeyStores (JKS or PKCS12). Ensure your keystore/truststore is in the correct format.

• If your organization uses OpenSSL-based tools, convert the PEM files (nifi.crt + nifi.key + CA chain) into a PKCS12/JKS keystore.

Problem 3: SAN Entries Not Recognized

• Verify the SAN entries in your certificate match the NiFi node hostnames (e.g. nifinode1.x.x.net).

• Use openssl x509 -in nifi.crt -text -noout to check SANs.

3. Step-by-Step Solution

A. Prepare the Keystore

1. Combine Certificate and Private Key:

   • If you have nifi.crt and nifi.key, create a PKCS12 keystore:

Spoiler

openssl pkcs12 -export \
-in nifi.crt \
-inkey nifi.key \
-chain -CAfile ca_chain.crt \ # Include CA chain if needed
-name "nifi" \
-out nifi-keystore.p12
-password pass:keystorepassword

1. • Use a password (e.g., keystorePassword).

2. Convert to JKS (if required) Java Keystore from PKCS12

Spoiler

keytool -importkeystore \
-srckeystore nifi-keystore.p12 \
-srcstoretype PKCS12 \
-destkeystore nifi-keystore.jks \
-deststoretype JKS

B. Prepare the Truststore

1. Import CA Certificates:

   • If your organization uses a private CA, add its root/intermediate certificates to the truststore

Spoiler

keytool -import -trustcacerts \
-alias ca-root \
-file ca_root.crt \
-keystore nifi-truststore.jks

C. Configure NiFi

1. Update nifi.properties

   Spoiler

   # HTTPS Settings
   nifi.web.https.host=0.0.0.0
   nifi.web.https.port=9443
   nifi.web.https.network.interface.default=

   # Security Properties #
   nifi.security.keystore=/path/to/keystore.jks nifi.security.keystoreType=JKS nifi.security.keystorePasswd=keystorepassword nifi.security.keyPasswd=keystorepassword nifi.security.truststore=/path/to/truststore.jks nifi.security.truststoreType=JKS nifi.security.truststorePasswd=truststorepassword

   # Truststore (required for cluster nodes/ZooKeeper)
   nifi.security.truststore=./nifi-truststore.jks
   nifi.security.truststoreType=JKS
   nifi.security.truststorePasswd=truststorePassword

   # Enable TLS for cluster nodes
   nifi.cluster.protocol.is.secure=true
   nifi.web.http.port=
   nifi.web.https.port=9443
   nifi.zookeeper.connect.string=zookeepernode1.x.x.net:2181,zookeepernode2.x.x.net:2181,zookeepernode3.x.x.net:2181
   nifi.zookeeper.client.secure=true

2. Updateauthorizers.xml (for mutual TLS): Configure for Cluster Communication

Configure authorizers.xml and nifi-registry.properties for secure cluster communication using the same certificates.

Spoiler

<property name="Initial Admin Identity">CN=admin, OU=YourOrg</property>
<property name="Node Identity 1">CN=nifinode1.x.x.net, OU=YourOrg</property>

D. Validate the Setup
After configuration

1. Test Keystore/Truststore:

Spoiler

# Verify keystore contents
keytool -list -v -keystore keystore.jks -storepass keystorepassword

# Verify truststore contents
keytool -list -v -keystore truststore.jks -storepass truststorepassword

# Test SSL configuration
openssl s_client -connect nifi-dev.x.x.net:9443 -showcerts

Troubleshooting Common Issues

1. Certificate Chain Issues: Ensure your keystore includes the full certificate chain
   Spoiler

   # Concatenate certificates if needed
   cat nifi.crt intermediate.crt root.crt > fullchain.crt
2. SAN Validation: Verify certificate has correct SAN entries
   Spoiler

   openssl x509 -in nifi.crt -text -noout | grep -A1 "Subject Alternative Name"
3. Java Compatibility: Ensure Java version compatibility with TLS
   Spoiler

   # Add to bootstrap.conf if using older Java versions java.arg.16=-Dhttps.protocols=TLSv1.2
4. Cluster Communication: Set proper node identities for cluster
   Spoiler

   # In nifi.properties
   nifi.cluster.node.address=nifinode1.x.x.net
   nifi.cluster.node.protocol.port=11443
   nifi.remote.input.secure=true
5. ZooKeeper Security: Only if using secure ZooKeeper connections
   Spoiler

   # In zookeeper.properties
   secureClientPort=2281 serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory ssl.keyStore.location=/path/to/keystore.jks ssl.keyStore.password=keystorepassword ssl.trustStore.location=/path/to/truststore.jks ssl.trustStore.password=truststorepassword

1. Verify HTTPS Access:

   • Access https://nifi-dev.x.x.net:9443/nifi in a browser.

   • Use curl -vk https://nifi-dev.x.x.net:9443/nifi to debug TLS handshake errors.

       2. Troubleshooting

• "SSL Peer Unauthenticated":

  • Ensure the truststore contains the CA certificate that signed the NiFi certificate.

• "Certificate Doesn't Match Hostname":

  • Verify SAN entries in nifi.crt include all NiFi node hostnames.

• Keystore Password Mismatch:

  • Ensure nifi.security.keystorePasswd and nifi.security.keyPasswd match in nifi.properties.

Additional Recommendations

1. Use strong, unique passwords for keystores and truststores
2. Implement proper certificate rotation procedures
3. Consider automating certificate management with tools like cert-manager
4. Implement client certificate authentication for additional security
5. Ensure proper DNS resolution for all SAN entries

By following the above steps, you’ll enable HTTPS for NiFi with proper SAN support and resolve keystore/truststore issues

Happy hadooping

[#3 · Accepted solution]

@pavanshettyg5 
The TLS implementation for NiFi requires proper configuration of both keystore and truststore your organization's signed certificate with SAN entries to enable secure HTTPS access. Based on your description, you've received a signed certificate (nifi.crt) but are experiencing issues with the complete TLS setup.
 Required Certificates & Files

• Certificate Authority (CA) Certificate: The root certificate from your signing authority
• Truststore: Contains the Certificate Authority (CA) root/intermediate certificates that signed your NiFi certificate (for mutual TLS or cluster communication).
• Signed Certificate (nifi.crt): Your domain certificate with the SAN entries
• Private Key: The private key used to generate the CSR

1. Prepare Certificate Files

Ensure you have:

• The signed certificate (nifi.crt)
• Your private key
• The CA certificate (request from your CA if not available)

Problem 1: Missing Private Key or Certificate Chain

• If you only have nifi.crt, you must also have:

  • The private key (e.g. nifi.key) generated during the CSR process.

  • The CA root/intermediate certificates (if your organization uses a private CA).

Problem 2: Improper Keystore/Truststore Format

• NiFi uses Java KeyStores (JKS or PKCS12). Ensure your keystore/truststore is in the correct format.

• If your organization uses OpenSSL-based tools, convert the PEM files (nifi.crt + nifi.key + CA chain) into a PKCS12/JKS keystore.

Problem 3: SAN Entries Not Recognized

• Verify the SAN entries in your certificate match the NiFi node hostnames (e.g. nifinode1.x.x.net).

• Use openssl x509 -in nifi.crt -text -noout to check SANs.

3. Step-by-Step Solution

A. Prepare the Keystore

1. Combine Certificate and Private Key:

   • If you have nifi.crt and nifi.key, create a PKCS12 keystore:

Spoiler

openssl pkcs12 -export \
-in nifi.crt \
-inkey nifi.key \
-chain -CAfile ca_chain.crt \ # Include CA chain if needed
-name "nifi" \
-out nifi-keystore.p12
-password pass:keystorepassword

1. • Use a password (e.g., keystorePassword).

2. Convert to JKS (if required) Java Keystore from PKCS12

Spoiler

keytool -importkeystore \
-srckeystore nifi-keystore.p12 \
-srcstoretype PKCS12 \
-destkeystore nifi-keystore.jks \
-deststoretype JKS

B. Prepare the Truststore

1. Import CA Certificates:

   • If your organization uses a private CA, add its root/intermediate certificates to the truststore

Spoiler

keytool -import -trustcacerts \
-alias ca-root \
-file ca_root.crt \
-keystore nifi-truststore.jks

C. Configure NiFi

1. Update nifi.properties

   Spoiler

   # HTTPS Settings
   nifi.web.https.host=0.0.0.0
   nifi.web.https.port=9443
   nifi.web.https.network.interface.default=

   # Security Properties #
   nifi.security.keystore=/path/to/keystore.jks nifi.security.keystoreType=JKS nifi.security.keystorePasswd=keystorepassword nifi.security.keyPasswd=keystorepassword nifi.security.truststore=/path/to/truststore.jks nifi.security.truststoreType=JKS nifi.security.truststorePasswd=truststorepassword

   # Truststore (required for cluster nodes/ZooKeeper)
   nifi.security.truststore=./nifi-truststore.jks
   nifi.security.truststoreType=JKS
   nifi.security.truststorePasswd=truststorePassword

   # Enable TLS for cluster nodes
   nifi.cluster.protocol.is.secure=true
   nifi.web.http.port=
   nifi.web.https.port=9443
   nifi.zookeeper.connect.string=zookeepernode1.x.x.net:2181,zookeepernode2.x.x.net:2181,zookeepernode3.x.x.net:2181
   nifi.zookeeper.client.secure=true

2. Updateauthorizers.xml (for mutual TLS): Configure for Cluster Communication

Configure authorizers.xml and nifi-registry.properties for secure cluster communication using the same certificates.

Spoiler

<property name="Initial Admin Identity">CN=admin, OU=YourOrg</property>
<property name="Node Identity 1">CN=nifinode1.x.x.net, OU=YourOrg</property>

D. Validate the Setup
After configuration

1. Test Keystore/Truststore:

Spoiler

# Verify keystore contents
keytool -list -v -keystore keystore.jks -storepass keystorepassword

# Verify truststore contents
keytool -list -v -keystore truststore.jks -storepass truststorepassword

# Test SSL configuration
openssl s_client -connect nifi-dev.x.x.net:9443 -showcerts

Troubleshooting Common Issues

1. Certificate Chain Issues: Ensure your keystore includes the full certificate chain
   Spoiler

   # Concatenate certificates if needed
   cat nifi.crt intermediate.crt root.crt > fullchain.crt
2. SAN Validation: Verify certificate has correct SAN entries
   Spoiler

   openssl x509 -in nifi.crt -text -noout | grep -A1 "Subject Alternative Name"
3. Java Compatibility: Ensure Java version compatibility with TLS
   Spoiler

   # Add to bootstrap.conf if using older Java versions java.arg.16=-Dhttps.protocols=TLSv1.2
4. Cluster Communication: Set proper node identities for cluster
   Spoiler

   # In nifi.properties
   nifi.cluster.node.address=nifinode1.x.x.net
   nifi.cluster.node.protocol.port=11443
   nifi.remote.input.secure=true
5. ZooKeeper Security: Only if using secure ZooKeeper connections
   Spoiler

   # In zookeeper.properties
   secureClientPort=2281 serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory ssl.keyStore.location=/path/to/keystore.jks ssl.keyStore.password=keystorepassword ssl.trustStore.location=/path/to/truststore.jks ssl.trustStore.password=truststorepassword

1. Verify HTTPS Access:

   • Access https://nifi-dev.x.x.net:9443/nifi in a browser.

   • Use curl -vk https://nifi-dev.x.x.net:9443/nifi to debug TLS handshake errors.

       2. Troubleshooting

• "SSL Peer Unauthenticated":

  • Ensure the truststore contains the CA certificate that signed the NiFi certificate.

• "Certificate Doesn't Match Hostname":

  • Verify SAN entries in nifi.crt include all NiFi node hostnames.

• Keystore Password Mismatch:

  • Ensure nifi.security.keystorePasswd and nifi.security.keyPasswd match in nifi.properties.

Additional Recommendations

1. Use strong, unique passwords for keystores and truststores
2. Implement proper certificate rotation procedures
3. Consider automating certificate management with tools like cert-manager
4. Implement client certificate authentication for additional security
5. Ensure proper DNS resolution for all SAN entries

By following the above steps, you’ll enable HTTPS for NiFi with proper SAN support and resolve keystore/truststore issues

Happy hadooping

[#4 · Accepted solution]

@pavanshettyg5 

What version of Apache NiFi are you using?

The NiFi screenshot you shared implies authentication was successful, but you are having some form of authorization issue.  The second screenshot you shared from the logs is not providing much useful information.

What is observed in both the nifi-user.log and nifi-app.log when you attempt to access the NiFi UI?

You mention that you are using "OIDC provider".  So when you access NiFi are you getting to the login prompt where you provide your OIDC credentials?

What is seen in the logs at this time and when you submit your credentials?

Does your NiFi truststore contain the complete trust chain (all root and intermediate public certs used to sign the server certificate) for your OIDC endpoint?

Please help our community grow and thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

[#5]

Hi @Shelton Thanks so much for detailed information
@MattWho  thanks much for the reply and apologies for short info. 
based on above information was able to create SSL certificates and generate Keystore and trustore in jks format . initially i was not configured CA file into truststore so faced some issue
2. then i did not added nifi nodes entries as intial identity in autherizers.xml file so above issue occured . i followed cloudera blogs where you had informed https://community.cloudera.com/t5/Support-Questions/insufficient-permissions-untrusted-proxy/m-p/366... 
based on these i was able to resolve and 3 node cluster with external zookeeper was able to up. 
i appreciate your kind help and your time here . much thanks to both 🙂