Member since
02-01-2022
293
Posts
105
Kudos Received
61
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 321 | 07-28-2026 12:03 PM | |
| 1455 | 05-15-2025 05:45 AM | |
| 5626 | 06-12-2024 06:43 AM | |
| 8780 | 04-12-2024 06:05 AM | |
| 6642 | 12-07-2023 04:50 AM |
06-01-2023
08:00 AM
3 Kudos
In this article I am going to review the required steps and processes to setup some NiFi SSL Context Services with modern versions of NiFi (1.20, 1.21, 2.0). In the past, nifi installations did not come installed with SSL enabled. If you were a NiFi admin and had to setup ssl, you know it was not an easy task, and could often times prevented the cluster from even being secured at all. Thanks to the wonderful team of innovators working on the Apache NiFi Project, I am happy to show the Easy Button works to install a fully secured Nifi. I will also show how to setup SSL Context Services for internal and external connection to https enabled endpoints. First, let's do a new install of Apache NiFi 1.21. This article is not about how to do that, so fast forward to a running NiFi, lets take a look at some important details that you will find in the nifi-app.log: Your login details: 2023-06-01 10:02:55,493 INFO [main] o.a.n.a.s.u.SingleUserLoginIdentityProvider
Generated Username [9cd754fa-9ca2-49f2-a627-53934a6876d6]
Generated Password [QQtZMerKoc9zmsGIT3X33OweIfm+nAd4]
Where you can find the NiFi UI: 2023-06-01 10:05:29,806 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:
2023-06-01 10:05:29,806 INFO [main] org.apache.nifi.web.server.JettyServer https://127.0.0.1:8443/nifi We should be able to now login to the NiFi UI on the secured host https://[nifihost]:8443 with the provided username and password. WOW: we have a user/pass to force a login right out of the box!! Alright, now let's get started with that internal SSL Context Service. We want to create the SSL Context Service Controller Service on the root nifi canvas. This will make the ssl context service available to all our process groups. In my last article Operationalize NiFi data flows with Cloudera DataFlow , I was connecting to https://[nifihost] endpoints from within NiFi to communicate with the NiFi REST API. This process should be similar with java cacerts, or your own custom keystore(s) and truststore(s). Let's find the SSL details used to secure NIFI in conf/nifi.properties: # security properties #
nifi.sensitive.props.key=7rOfiLY584X8nNpYMdye6p2DjwfgrvW3
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.additional.keys=
nifi.security.autoreload.enabled=false
nifi.security.autoreload.interval=10 secs
nifi.security.keystore=./conf/keystore.p12
nifi.security.keystoreType=PKCS12
nifi.security.keystorePasswd=1391fbf8ada209439bd99b95432892ca
nifi.security.keyPasswd=1391fbf8ada209439bd99b95432892ca
nifi.security.truststore=./conf/truststore.p12
nifi.security.truststoreType=PKCS12
nifi.security.truststorePasswd=0a85b23929af71e49990916bb73e1733
nifi.security.user.authorizer=single-user-authorizer
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=single-user-provider
nifi.security.user.jws.key.rotation.period=PT1H
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate= Now that we have the details we need (keystore,truststore,and passwords) we can make a new SSL Context Service like this: It is still possible to create more SSL Context Services, especially if you have self signed or custom certs attached to external endpoints you need to communicate with. To connect to most public signed certs, java's cacerts works great. This is always the first SSL Context Service I start with for connecting to public https:// endpoints. First I need to copy cacerts to my nifi conf directory: cp /usr/lib/jvm/java-11-openjdk-11.0.13.0.8-4.el8_5.x86_64/lib/security/cacerts /root/nifi-1.21.0/conf/ Now I can create a new SSL Context Service like this: The cacerts password is "changeit". In conclusion, installing a fully secured NIFI with basic user auth and SSL is now much easier than in older versions of NiFi. Additionally when setting up NiFi SSL Context Service(s) just be sure to get all the right details and they will work as expected. If you land on this article and are still struggling with setting up your own SSL Context Service, create a new community post here and give me an @steven-matison and I will be glad to help out!!
... View more
Labels:
06-01-2023
06:31 AM
1 Kudo
@drewski7 I think the solution you are looking for is to use one of the alternative data stores that lives outside of NiFi with the map cache. The options are Redis, Hbase, Cassandra, Couchbase: They will give you greater control and they are preferred for production and large volumes.
... View more
05-31-2023
07:29 AM
Cloudera making hard stuff easy again! Great article Ryan!!
... View more
05-31-2023
07:26 AM
@jnifi It sounds like you mentioned missing some headers. If you know what they are, you click the + on nifi invokeHttp and add them. If you get into debug, and the log files you should be able to see the request and response objects to confirm its the correct format. I think that will get you through the 401.
... View more
05-31-2023
05:35 AM
2 Kudos
@Fredi The solution you are looking for, is to use this line of code in your script during a loop whenever you want to send the current flowfile content. session.commit() This will send a flowfile out. This is little known, as this command is inferred at the end of the script, assuming 1 execution to one flowfile. Here is an example from my fraud detection demo, a while statement that does some counts, sends multiple flowfiles of good transactions, then in random iterations during 20 loops, sends some fraud transaction flowfiles. # All processing code starts at this indent
while ticks < 20:
ticks += 1
fintran = create_fintran()
fintransaction = json.dumps(fintran)
#send_fintran(out_socket, json.dumps(fintran))
#print(fintransaction)
flowFile = session.create()
flowFile = session.write(flowFile, WriteContentCallback(fintransaction))
session.transfer(flowFile, REL_SUCCESS)
session.commit()
sleep(DELAY)
if ticks > fraud_tick:
fraudtran = create_fraudtran(fintran)
fraudfintransaction=json.dumps(fraudtran)
#send_fintran(out_socket, json.dumps(fraudtran))
#print(fraudfintransaction)
flowFile2 = session.create()
flowFile2 = session.write(flowFile2, WriteContentCallback(fraudfintransaction))
session.transfer(flowFile2, REL_SUCCESS)
session.commit()
fraud_tick = random.randint(FRAUD_TICK_MIN, FRAUD_TICK_MAX)
... View more
05-31-2023
04:38 AM
Pull Request is in!! https://github.com/apache/nifi/pull/7316 This will be fixed in future releases of NiFi. @jisaitua send me a DM with your email if you need to use the new processor right away.
... View more
05-30-2023
11:42 AM
1 Kudo
@jisaitua I am not sure how long its going to take, but i should have a github commit ready w/ a new PutBigQuery nar that you can use with 1.21. I will update again tomorrow morning.
... View more
05-30-2023
11:05 AM
Some progress: resource 'projects/gcp-se-cdp-sandbox-env/datasets/dataset/tables/tablename' With datasets = ${bq.dataset} tables = ${bq.table.name}
... View more
05-30-2023
05:58 AM
1 Kudo
@jisaitua Thank you for making such a well written post with all of the right screen shots, etc. I am working on duplicating this issue and so far i have confirmed the same experience. INVALID_ARGUMENT: Invalid project resource name projects/${test}; Project id: ${test} So this is a known JIRA bug: https://issues.apache.org/jira/projects/NIFI/issues/NIFI-11608?filter=allissues I will work on getting some traction here!!
... View more
05-30-2023
05:31 AM
@jnifi If you are able to get a remote API call working with Postman, you can definitely get that to work with InvokeHttp. Not having an oAuth2 to test with myself, here are some suggestions: Fully document required settings for postman headers, request, post values, etc duplicate them in NiFi set InvokeHttp log level to DEBUG tail logs/nifi-app.log while testing When you have that operational understanding and full visibility of errors begin iterating on NiFi where you start to test all of the settings. Getting nifi/invokeHttp outcome to match postman is much easier in this manner. Additionally in future replies, show screen shots of processor configs or share flow definition/template files where possible.
... View more