Support Questions

mslnrd · ‎08-01-2023

Hello,

I am trying to setup a HTTPS on NiFi Registry and am receiving SSL_ERROR_BAD_CERT_ALERT when attempting to access the URL after loading the certificates.

The way I am setting it up is that I create a CSR, have the CA sign it (AD Domain Controller) and import that certificate into the keystore.

I have generated the keystore, the truststore imported the certificates, and when I restart the NiFi Registry service the logs show no errors so I am not sure what I am doing wrong.

Below are some sanitized outputs of some commands I've ran to show what I could potentially have done wrong.

Curl Output:

curl --cert-type P12 --cert ./keystore.p12:Password --cacert /root/ssl/ADcerts/ad1-ca.cer -v https://nifiregistry.our.network.com:8443/nifi-registry
*   Trying 123.4.56.78:8443...
* Connected to nifiregistry.our.network.com (123.4.56.78) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /root/ssl/ADcerts/ad1-ca.cer
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=Country; ST=State; L=City; O=Org; OU=OrgUnit; CN=nifiregistry
*  start date: Jul 31 20:04:43 2023 GMT
*  expire date: Jul  6 17:37:31 2038 GMT
*  subjectAltName: host "nifiregistry.our.network.com" matched cert's "nifiregistry.our.network.com"
*  issuer: DC=com; DC=network; DC=our; CN=AD1-CA
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET /nifi-registry HTTP/1.1
> Host: nifiregistry.our.network.com:8443
> User-Agent: curl/7.76.1
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Tue, 01 Aug 2023 17:20:26 GMT
< Content-Security-Policy: frame-ancestors 'self'
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31540000
< Location: https://nifiregistry.our.network.com:8443/nifi-registry
< Content-Length: 0
<
* Connection #0 to host nifiregistry.our.network.com left intact

Keystore -V

curl --cert-type P12 --cert ./keystore.p12:Password --cacert /root/ssl/ADcerts/ad1-ca.cer -v https://nifiregistry.our.network.com:8443/nifi-registry
*   Trying 123.4.56.78:8443...
* Connected to nifiregistry.our.network.com (123.4.56.78) port 8443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /root/ssl/ADcerts/ad1-ca.cer
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS header, Unknown (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=Country; ST=State; L=City; O=Org; OU=OrgUnit; CN=nifiregistry
*  start date: Jul 31 20:04:43 2023 GMT
*  expire date: Jul  6 17:37:31 2038 GMT
*  subjectAltName: host "nifiregistry.our.network.com" matched cert's "nifiregistry.our.network.com"
*  issuer: DC=com; DC=network; DC=our; CN=AD1-CA
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Unknown (23):
> GET /nifi-registry HTTP/1.1
> Host: nifiregistry.our.network.com:8443
> User-Agent: curl/7.76.1
> Accept: */*
>
* TLSv1.2 (IN), TLS header, Unknown (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Unknown (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 302 Found
< Date: Tue, 01 Aug 2023 17:20:26 GMT
< Content-Security-Policy: frame-ancestors 'self'
< X-Frame-Options: SAMEORIGIN
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31540000
< Location: https://nifiregistry.our.network.com:8443/nifi-registry
< Content-Length: 0
<
* Connection #0 to host nifiregistry.our.network.com left intact

Nifi Properties

# web properties #
nifi.registry.web.war.directory=./lib
nifi.registry.web.http.host=
nifi.registry.web.http.port=
nifi.registry.web.https.host=123.4.56.78
nifi.registry.web.https.port=8443
nifi.registry.web.https.application.protocols=http/1.1
nifi.registry.web.jetty.working.directory=./work/jetty
nifi.registry.web.jetty.threads=200
nifi.registry.web.should.send.server.version=true

# security properties #
nifi.registry.security.keystore=./conf/keystore.p12
nifi.registry.security.keystoreType=PKCS12
nifi.registry.security.keystorePasswd=Password
nifi.registry.security.keyPasswd=Password
nifi.registry.security.truststore=./conf/truststore.p12
nifi.registry.security.truststoreType=PKCS12
nifi.registry.security.truststorePasswd=Password
nifi.registry.security.needClientAuth=
nifi.registry.security.authorizers.configuration.file=./conf/authorizers.xml
nifi.registry.security.authorizer=managed-authorizer
nifi.registry.security.identity.providers.configuration.file=./conf/identity-providers.xml
nifi.registry.security.identity.provider=ldap-identity-provider

mslnrd · ‎08-23-2023

@MattWho

I was able to get this resolved. It didn't require adding any other certificates other than my CA server certificate to the truststore for each server. They share the same CA and so I used the same certificate in each truststore.

First, I am not sure why, but, NiFi did not like using port 636 for LDAPS, so I set it up to use 3269 instead and the SAN error went away. Everything I tried prior did not resolve the error, it would give me an error that it couldn't find a SAN with my.network.com:636 along with partial LDAP result not being able to find my.network.com, which didn't make sense because all of the troubleshooting I did would come back good. I could even resolve SSL using my key entry from my truststore.

Once I had it up and working after that, I had to figure out why the buckets weren't loading. So I combed through a bunch of posts and found one that you wrote to someone having a similar issue and mentions the node needs to exist in NiFi-Registry. So, I added the node as "CN=node.hostname.com" (how I have it on the certificate) as a user account in NiFi Registry with the ability to read buckets and proxy user requests and bam buckets showed up.

Seems like everything is working great now. I appreciate your help.

View solution in original post

DianaTorres · ‎08-01-2023

@mslnrd Welcome to the Cloudera Community!

To help you get the best possible solution, I have tagged our NiFi expert @steven-matison who may be able to assist you further.

Please keep us updated on your post, and we hope you find a satisfactory solution to your query.

Regards,

Diana Torres,
Community Moderator

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:
Community Guidelines
How to use the forum

mslnrd · ‎08-02-2023

Hi,

Thanks for your response. I believe I may have figured out what the issue was.

I added false to

nifi.registry.security.needClientAuth=

since we are not using client authentication for users or systems.

MattWho · ‎08-02-2023

@mslnrd
The NiFi Registry configuration property:

nifi.registry.security.needClientAuth=

is used to control whether NiFi-registry whether the TLS exchange is mutual (2-way TLS handshake if set to true) or 1-way TLS handshake (when set to false).

At a very high level, in a mutual TLS handshake both the server and the client need to provide their certificate in the the TLS handshakes. When you changed this to false, NiFi-Registry no longer "required" that a client certificate is presented to identify the client who connected to the NiFi-Registry. If no other method of authentication is enabled, NiFi-Registry will treat the client as "anonymous". NiFi-Registry supports anonymous user access to the NiFi-Registry UI. This access is limited to accessing already created "public" buckets and ability to read flows version controlled in to a "public" bucket.

The ability to create buckets, administer buckets and flows, and write version controlled flows to any bucket would require an authenticated and authorized user.

So it is not clear how this simple change solved your issue. I think this change simple allowed the anonymous access only.

You'll need to inspect the keystore and truststore used by both NiFi (registry client ssl context) and NiFi-Registry (configured in nifi-registry.properties file) to make sure a mutual TLS exchange is possible.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

mslnrd · ‎08-03-2023

That would explain why I am seeing an error about an anonymous user now. Would this also be the reason that NiFi cannot see the buckets created in NiFi registry? Our setup NiFi and NiFi Registry on separate servers, with users authenticating through LDAPS (Active Directory) instead of using certificate authentication.

The unfortunate part is that I was handed off this project with NiFi and Registry already installed and "configured" so I am not sure exactly how far they went into the secure setup; or if they actually did it at all so, I have been trying to find time to work through the secure configurations.

When I change:

nifi.registry.security.needClientAuth=

to true I cannot even get the registry page to load, I see the error:

Error code: SSL_ERROR_BAD_CERT_ALERT and longs don't show any errors that I can see. I think the problem I am struggling with is that I can't find any documentation on how I should be setting up NiFi and the Registry with an internal CA and using LDAP for user authentication. The documentation that I find seems to be geared towards generating user certificates to configure SSL and authentication with both products on the same server.

MattWho · ‎08-04-2023

@mslnrd
Would not expect to see any error in the NiFi logs. The TLS exchange happens at a lower level and it is expected that if server (NiFi) is configured to REQUIRE a client certificate and one is not provided, the connection is closed.

The requirements for the certificate used by NiFi-Registry are no different then NiFi. The complete Certificate Authority (CA) chain for the client auth certificate in the NiFi-Registry must be present in NiFi's truststore.
Similarly the the complete Certificate Authority (CA) chain for the client auth certificate in the NiFi must be present in NiFi-Registry's truststore.

This is only way that the mutual TLS exchange/handshake will be successful negotiated.

While you can use the NiFi TLS toolkit to generate your certificates, that is not a requirement. You can use free services or generate your own certificates as long as the meet the requirements.

keystore must contain only one PrivateKeyEntry
The PrivateKeyEntry must have both ClientAuth and ServerAuth ExtendedKeyUsage (EKU)s.
The PrivateKeyEntry must include a Subject Alternative Name (SAN) entry matching the hostname of the server on which that certificate will be used for serverAuth. You can have multiple SANs as your server may be known by a hostname, IP, or secondary hostname.

The truststore will contain one too many TrustedCertEntries (public certificates).
Each TrustedCertEntry is for either an intermediate CA(s) or root CA(s). An intermediate CA would be any certificate authority where the owner is different than the issuer/signer of that public certificate. Root CAs will have same owner and issuer.

A complete trustchain means you have every CA leading from the issuer signer of your client certificate to the root CA. There may be 0 or more intermediate CAs between the client and rootCA.

The only way the NiFi and NiFi-Registry will authenticate with each other is through a mutual TLS exchange. The NiFi certificate DN(s) will need to be authorized within your NiFi-Registry and so will your NiFi user. The NiFi host(s) will proxy requests to NiFi-Registry on behalf of the authenticated user in NIFi. Even though the NiFi is acting as a proxy, the user will still need be properly authorized as the results form the proxy request will be for only the buckets and flows for which the proxied user is authorized against.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

mslnrd · ‎08-09-2023

@MattWho

Thanks for that information, I think I have it partially figured out now.

We only have 1 CA, so it is easier for me to sign certificates as needed while I work through this. So far this is what has been done:

Each Server (NiFi and NiFi Registry) using Keystore Explorer, I generated a keypair in the Keyststore with the required extensions (ClientAuth, ServerAuth & SANs)
Created a CSR and had it signed by the CA.
Imported the CA reply back into the Keystore with the full chain
In the Truststore, I imported the root certificate
Each server has LDAPS authentication setup, with the same default admin account that is able to login using it's AD credentials
Added the NiFi server to authorizations.xml for Node 1 using the full DN from AD

There is still something I am struggling with though:

The only way the NiFi and NiFi-Registry will authenticate with each other is through a mutual TLS exchange. The NiFi certificate DN(s) will need to be authorized within your NiFi-Registry and so will your NiFi user. The NiFi host(s) will proxy requests to NiFi-Registry on behalf of the authenticated user in NIFi. Even though the NiFi is acting as a proxy, the user will still need be properly authorized as the results form the proxy request will be for only the buckets and flows for which the proxied user is authorized against.

How would I go about doing this? I've tried importing each server's certificate into the others truststore but I started receiving errors about certificate path not being found and the most recent error is certificate_unknown.

MattWho · ‎08-10-2023

@mslnrd
Don't confuse authentication with authorization.
The only PrivateKeyEntry that can be in the NiFi-Registry keystore is the private key for the NiFi-Registry host on which NiFi-Registry is installed.
The only PrivateKeyEntry that can be in the NiFi keystore is the private key for the NiFi host on which NiFi- is installed.

These PrivateKeyEntries need to have EKUs for both clientAuth and ServerAuth.

The truststore can have 1 too many TrustedCertEntries. Typically you would maintain singled truststore you use everywhere that contains all the intermediate and root Certificate Authorities (CAs) used to sign/issue the privateKey. A truststore would not contain any PrivateKeyEntries. It is intended for only public certs.

When NiFi initiates a connection to the configured NiFi-Registry client, a TLS handshake occurs, within that handshake the NIFi-registry (server side of handshake) will want NiFi to identify itself via a certificate that NiFi-Registry is capable of trusting. The DistinguishedName (DN) of that clientAuth NiFi certificate is then used as the authenticated user identity. Also included in that exchange is the user identity for the user who is authenticated in NiFi initiating this client connection (via action of starting version control, stopping version control, initiating a version change, etc).

At that point authentication is done and authorization must then happen. NiFi-Registry will check to see that the NiFi clientAuth DN is authorized to proxy the NIFi user's request and verify that the NiFi user identity is authorized to read and/or write to the bucket(s) in NiFi-registry. These authorizations are setup from within the NiFi-registry UI.

The NiFi-Registry Admin guide covers the NiFi-Registry Access Policies and what those policies allow the authenticated user to do.

The NiFi-Registry User Guide covers how to manage users via the NiFi-Registry UI.

Since NiFi-Registry (just like NiFi) has identity mapping properties in the nifi-registry.properties configuration file to manipulate the user identity strings post successful authentication, the full DN may not be passed to the configured authorizer that handles the authorization phase. It is common to have mapping s setup to trim only the CN out of the full DN of a certificate. Keep in mind that the user identity string passed to the authorizer is case sensitive and must match exactly. The nifi-registry-app.log should be logging requests made by an authenticated user. If you are seeing "anonymous" user in that log, then authentication was nit successful. NiFi-Registry allows "anonymous" users to read from public buckets only.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

mslnrd · ‎08-18-2023

Sorry I haven't responded I've been busy working on this and can still say I haven't gotten it working.

Not sure what the problem is that I'm having. I even spun up a bran new Linux Server and put NiFi and NiFi Registry on the same server. Can't even NiFi to start with the certificates I've provided it.

All I receive is:

==> nifi-bootstrap.log <==
2023-08-18 16:34:30,290 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: Error creating bean with name 'niFiWebApiConfiguration': BeanPostProcessor before instantiation of bean failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.method.configuration.PrePostMethodSecurityConfiguration': Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.security.configuration.AuthenticationSecurityConfiguration': Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizer': FactoryBean threw exception on object creation; nested exception is org.springframework.ldap.PartialResultException: nested exception is javax.naming.PartialResultException [Root exception is javax.naming.CommunicationException: my.network.com:636 [Root exception is javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching my.network.com found.]]

Not really sure it is as difficult to do this as I am making it out to be but I can say it's frustrating.

If there are any guides that show step-by-step how to setup SSL with LDAPS authentication (not using any user certificates or requiring browsers to import their certificates) I'd appreciate it. Everything I can find is either setting it up with Ranger, Ambari, or using standalone certificates.

MattWho · ‎08-22-2023

@mslnrd
TLS/SSL is nothing specific to NiFi or LDAP. There are certain requirements in the TLS exchange specification. This applies to any application that would use TLS.

Based on your exception:

[Root exception is javax.naming.CommunicationException: my.network.com:636 [Root exception is javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching my.network.com found.]]

The issue happens during the TLS exchange. NiFi's ldap-user-group-provider within the authorizers.xml file is attempting to execute during NiFi startup in order to sync users and groups from your target LDAPS server.

At a very high basic level here, you have configured NiFi to connect to your LDAPS host "my.network.com". NiFi is acting as the client in the TLS exchange and the LDAPS as the server side. Your LDAPS is returning a serverAuth certificate that is missing a Subject Alternative Name (SAN) entry matching the hostname (my.network.com) the client is trying to connect with. Within the TLS spec, the client is then rejecting this connection as insecure since it can not verify the authenticity of the server that is responding. It assumes a man in the middle type issue where client is trying to establish a secure connection with server "my.network.com"; however, certificate returned does not indicate it belongs to that server via its list of SAN entries. So the issue here is with the certificate being used by your LDAPS server missing the expected SAN entry. NiFi does not provide an option to force allow an insecure connection.

Bottom line is the NiFi Keystore needs to meet following requirements:
1. The Keystore can contain only one PriavteKeyEntry
2. The PrivateKeyEntry must have both clientAuth and ServerAuth Extended Key Usage (EKUs)
3. The PrivateKeyEnrty must have a SAN entry that matches the hostname of the server on which NiFi is installled.

The NiFi Truststore needs to meet the following:

1. The truststore can contain one too many TrustedCert entries.

2. The complete trust chain for any server certificates NiFi will need to establish a TLS connection with will need to be present in the truststore. When a certificate is created it has an owner and a signer/issuer. The signer of the certificate might be self-signed (meaning owner DN and signer/issuer DN are the same), it might be an intermediate CA (Owner DN is different the signer/issuer DN), and a Root CA (also has matching owner DN and signer/issuer DN). A complete trust chain means all trusted certificates (public certs) from client certificate to root CA need to be present in the truststore. This could encompass the server public cert, one or more intermediate CA public certs, and the CA root public cert.

Often, but not always you can get all the public certificates using openssl.

openssl s_client -connect <target server hostname>:<port> -showcerts

Above would initiate a TLS handshake with target server and will return within the response the public certs from the target server. It may or may not have complete trustchain. You can then load these public certificates into your truststore.

In your case however, it does not look like a trust chain issue at this point. As i mentioned above, your current issue is that your serverAuth certificate used by your target LDAP is missing the TLS spec required SAN entry.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

Support Questions

NiFi Registry HTTPS Setup Giving SSL_ERROR_BAD_CERT_ALERT