Skip to main content
tcherian
tcherian
Silver ⭐️⭐️
June 11, 2024
Solved

How to access Nifi REST API 2.0.0 from a docker container

Hello there! 

I am currently running NiFi 2.0.0 M1 on my host machine and I want to use the NiFi REST API to start and stop processors for my Airflow tasks. Airflow is running inside a Docker container. However, I am facing issues when trying to access the NiFi REST API from within the container.

I attempted to access the NiFi API from the Airflow container using https://host.docker.internal:8443/nifi-api but I received an HTTP ERROR 400 with the message "Invalid SNI".

I also tried containerizing both NiFi 2.0.0 M1 and Airflow using Docker Compose and access the API endpoint using https://<nifi-container-name>:8443/nifi-api. This also resulted in an HTTP ERROR 400 with the "Invalid SNI" message.

I understand this is related to the SSL/TSL handshake and that from NiFi2.x default behaviour does not support accessing NiFi using IP addresses. However, I'm not sure how to resolve this. Please suggest a way to connect to NiFi from inside a container?

MattWho
MattWhoAnswer
Titanium ⭐️⭐️⭐️
June 13, 2024

@tcherian 

NiFi certificates must meet the following criteria:

1.  No wildcards used in the subject DistinquishedName (DN) 
2. Included both clientAuth and serverAuth in the ExtendedKeyUsage (EKU)
3. Contains one or more SubjectAlternativeName (SAN) entries.
4. Keystore can only contain 1 PrivateKey entry

There are many resources on the web for generating your own self-signed certificates and adding them to a PKCS12 or JKS keystore.  The "Keystore" and "truststore" are both just keystores.  The NiFi "Keystore" contain the PrivateKey entry which Is used by NiFi to identify itself as the server (serverAuth) when connecting to it and as the client (clientAuth) when connecting outward as a client (such as talking to other NiFi's, NiFi-Registry, etc).  The NiFi "truststore" contains one too many TrustedCert entries.  It is common to use the default Java cacerts file (which is just a jks keystore) and add additional TrustedCert entries to it.  The trustedCerts are the public certs that correspond to the PrivateKey that you should never share.  The Trusted certs are the signers of the private keys.  There are intermediate and root trusted cert keys.  An intermediate trust is one where the owner and signer are not the same DN.  A root trust is one where the owner and signer are the same DN.   So you might create a PrivateKey that is signed by intermediate Certificate Authority (CA) and that intermediate CA would be signed by another intermediate CA or a root (CA). The chain of signers between intermediate and root is known as the trustchain.  The Truststore needs to contain complete trust chains for your PrivateKey.   

There are even free services out there like Tinycert, but you can also use openssl and keystool to generate self-signed certificates and import them to a keystore.  Just google how to create a certificate and how to import certificate into a keystore.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

View in thread

4 replies

MattWho
Titanium ⭐️⭐️⭐️
MattWho
Titanium ⭐️⭐️⭐️
June 11, 2024

@tcherian 
I assume you are using the non-production ready NiFi out-of-the-box auto-generated keystore and truststore keystores files?

If so, you should generate your own certificates that include the additional "host.docker.internal" and/or "nifi-container-name" SAN entries.   Import that certificate into your own keystore and populated a truststore with the complete trust chain for your certificate.

Something else you might want to try is to populate the the following property in the nifi.properties file:

nifi.web.proxy.host=host.docker.internal,nifi-container-name


But even if above works for you, i would still highly encourage you to get actual signed certificates instead.

Please help our community thrive. If you found any of the suggestions/solutions provided helped you with solving your issue or answering your question, please take a moment to login and click "Accept as Solution" on one or more of them that helped.

Thank you,
Matt

Translate
tcherian
tcherianAuthor
Silver ⭐️⭐️
June 13, 2024

Hello @MattWho 

Thanks for the response.

You were correct in assuming that I am using NiFi with the out-of-the-box auto-generated keystore and truststore files. I am also not running NiFi in a cluster but in a  standalone instance.

I tried to tried to changing the nifi.web.proxy.host in the nifi.properties file as below but  it didn’t work for me.

# web properties #
#############################################

# For security, NiFi will present the UI on 127.0.0.1 and only be accessible through this loopback interface.
# Be aware that changing these properties may affect how your instance can be accessed without any restriction.
# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.

nifi.web.http.host=
nifi.web.http.port=
nifi.web.http.network.interface.default=

#############################################

nifi.web.https.host=127.0.0.1
nifi.web.https.port=8443
nifi.web.https.network.interface.default=
nifi.web.https.application.protocols=h2 http/1.1
nifi.web.jetty.working.directory=./work/jetty
nifi.web.jetty.threads=200
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=
nifi.web.proxy.host=host.docker.internal,nifi
nifi.web.max.content.size=
nifi.web.max.requests.per.second=30000
nifi.web.max.access.token.requests.per.second=25
nifi.web.request.timeout=60 secs
nifi.web.request.ip.whitelist=
nifi.web.should.send.server.version=true
nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i" "%{User-Agent}i"

When I tried accessing https://host.docker.internal:8443/nifi-api/access/token from my airflow container(Nifi was running in host machine at localhost:8443), it returned a Error 400 Invalid SNI.

<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 400 Invalid SNI</title>
</head>
<body><h2>HTTP ERROR 400 Invalid SNI</h2>
<table>
<tr><th>URI:</th><td>/nifi-api/access/token</td></tr>
<tr><th>STATUS:</th><td>400</td></tr>
<tr><th>MESSAGE:</th><td>Invalid SNI</td></tr>
<tr><th>SERVLET:</th><td>-</td></tr>
<tr><th>CAUSED BY:</th><td>org.eclipse.jetty.http.BadMessageException: 400: Invalid SNI</td></tr>
</table>
<h3>Caused by:</h3><pre>org.eclipse.jetty.http.BadMessageException: 400: Invalid SNI
        at org.eclipse.jetty.server.SecureRequestCustomizer.customize(SecureRequestCustomizer.java:266)
        at org.eclipse.jetty.server.SecureRequestCustomizer.customize(SecureRequestCustomizer.java:207)
        at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1594)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
        at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:558)
        at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:379)
        at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:146)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
        at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:1583)
</pre>

</body>
</html>
Translate
    Terms & ConditionsAccessibility statement