Member since
02-01-2022
274
Posts
97
Kudos Received
60
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
439 | 05-15-2025 05:45 AM | |
3470 | 06-12-2024 06:43 AM | |
6106 | 04-12-2024 06:05 AM | |
4174 | 12-07-2023 04:50 AM | |
2243 | 12-05-2023 06:22 AM |
06-05-2023
06:30 AM
@drewski7 The two posts below have solutions to query the json: https://community.cloudera.com/t5/Community-Articles/Running-SQL-on-FlowFiles-using-QueryRecord-Processor-Apache/ta-p/246671 https://community.cloudera.com/t5/Support-Questions/QueryRecord-processor-issue-with-nested-JSON/td-p/338556 Here a the NiFI doc with more high level SQL info: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/latest/org.apache.nifi.processors.standard.QueryRecord/additionalDetails.html You can find other examples here in the community just search something like: queryrecord + json.
... View more
06-05-2023
06:14 AM
@Dracile If you are looking to iterate through the results inside of your upstream json object, you need QueryRecord with a Json Reader and Writer. This allows you to provide the upstream schema (reader), downstream schema (writer) and a query against the flowfile. This will unfortunately lose the original object values You can find an example here: https://github.com/cldr-steven-matison/NiFi-Templates/blob/main/QueryRecord_Sample.json You will need to modify the json object in GenerateFlowFile, then adjust the Reader/Writer, and the results query into $.VisitList[] array. Once you have this lil mini test working, take the logic to your final flow.
... View more
06-01-2023
12:40 PM
2 Kudos
@drewski7 You are looking for something like this, for "yesterday": ${now():minus(86400000):format('MM-dd-yyyy hh:mm:ss') } Then change 24 hours to 10 hours and add not reduce: ${now():plus(36000000):format('MM-dd-yyyy hh:mm:ss') } Let me know if this adapts to fit your use case!
... View more
06-01-2023
08:00 AM
3 Kudos
In this article I am going to review the required steps and processes to setup some NiFi SSL Context Services with modern versions of NiFi (1.20, 1.21, 2.0). In the past, nifi installations did not come installed with SSL enabled. If you were a NiFi admin and had to setup ssl, you know it was not an easy task, and could often times prevented the cluster from even being secured at all. Thanks to the wonderful team of innovators working on the Apache NiFi Project, I am happy to show the Easy Button works to install a fully secured Nifi. I will also show how to setup SSL Context Services for internal and external connection to https enabled endpoints. First, let's do a new install of Apache NiFi 1.21. This article is not about how to do that, so fast forward to a running NiFi, lets take a look at some important details that you will find in the nifi-app.log: Your login details: 2023-06-01 10:02:55,493 INFO [main] o.a.n.a.s.u.SingleUserLoginIdentityProvider
Generated Username [9cd754fa-9ca2-49f2-a627-53934a6876d6]
Generated Password [QQtZMerKoc9zmsGIT3X33OweIfm+nAd4]
Where you can find the NiFi UI: 2023-06-01 10:05:29,806 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:
2023-06-01 10:05:29,806 INFO [main] org.apache.nifi.web.server.JettyServer https://127.0.0.1:8443/nifi We should be able to now login to the NiFi UI on the secured host https://[nifihost]:8443 with the provided username and password. WOW: we have a user/pass to force a login right out of the box!! Alright, now let's get started with that internal SSL Context Service. We want to create the SSL Context Service Controller Service on the root nifi canvas. This will make the ssl context service available to all our process groups. In my last article Operationalize NiFi data flows with Cloudera DataFlow , I was connecting to https://[nifihost] endpoints from within NiFi to communicate with the NiFi REST API. This process should be similar with java cacerts, or your own custom keystore(s) and truststore(s). Let's find the SSL details used to secure NIFI in conf/nifi.properties: # security properties #
nifi.sensitive.props.key=7rOfiLY584X8nNpYMdye6p2DjwfgrvW3
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.additional.keys=
nifi.security.autoreload.enabled=false
nifi.security.autoreload.interval=10 secs
nifi.security.keystore=./conf/keystore.p12
nifi.security.keystoreType=PKCS12
nifi.security.keystorePasswd=1391fbf8ada209439bd99b95432892ca
nifi.security.keyPasswd=1391fbf8ada209439bd99b95432892ca
nifi.security.truststore=./conf/truststore.p12
nifi.security.truststoreType=PKCS12
nifi.security.truststorePasswd=0a85b23929af71e49990916bb73e1733
nifi.security.user.authorizer=single-user-authorizer
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=single-user-provider
nifi.security.user.jws.key.rotation.period=PT1H
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate= Now that we have the details we need (keystore,truststore,and passwords) we can make a new SSL Context Service like this: It is still possible to create more SSL Context Services, especially if you have self signed or custom certs attached to external endpoints you need to communicate with. To connect to most public signed certs, java's cacerts works great. This is always the first SSL Context Service I start with for connecting to public https:// endpoints. First I need to copy cacerts to my nifi conf directory: cp /usr/lib/jvm/java-11-openjdk-11.0.13.0.8-4.el8_5.x86_64/lib/security/cacerts /root/nifi-1.21.0/conf/ Now I can create a new SSL Context Service like this: The cacerts password is "changeit". In conclusion, installing a fully secured NIFI with basic user auth and SSL is now much easier than in older versions of NiFi. Additionally when setting up NiFi SSL Context Service(s) just be sure to get all the right details and they will work as expected. If you land on this article and are still struggling with setting up your own SSL Context Service, create a new community post here and give me an @steven-matison and I will be glad to help out!!
... View more
Labels:
06-01-2023
06:31 AM
1 Kudo
@drewski7 I think the solution you are looking for is to use one of the alternative data stores that lives outside of NiFi with the map cache. The options are Redis, Hbase, Cassandra, Couchbase: They will give you greater control and they are preferred for production and large volumes.
... View more
05-31-2023
07:29 AM
Cloudera making hard stuff easy again! Great article Ryan!!
... View more
05-31-2023
07:26 AM
@jnifi It sounds like you mentioned missing some headers. If you know what they are, you click the + on nifi invokeHttp and add them. If you get into debug, and the log files you should be able to see the request and response objects to confirm its the correct format. I think that will get you through the 401.
... View more
05-31-2023
05:35 AM
2 Kudos
@Fredi The solution you are looking for, is to use this line of code in your script during a loop whenever you want to send the current flowfile content. session.commit() This will send a flowfile out. This is little known, as this command is inferred at the end of the script, assuming 1 execution to one flowfile. Here is an example from my fraud detection demo, a while statement that does some counts, sends multiple flowfiles of good transactions, then in random iterations during 20 loops, sends some fraud transaction flowfiles. # All processing code starts at this indent
while ticks < 20:
ticks += 1
fintran = create_fintran()
fintransaction = json.dumps(fintran)
#send_fintran(out_socket, json.dumps(fintran))
#print(fintransaction)
flowFile = session.create()
flowFile = session.write(flowFile, WriteContentCallback(fintransaction))
session.transfer(flowFile, REL_SUCCESS)
session.commit()
sleep(DELAY)
if ticks > fraud_tick:
fraudtran = create_fraudtran(fintran)
fraudfintransaction=json.dumps(fraudtran)
#send_fintran(out_socket, json.dumps(fraudtran))
#print(fraudfintransaction)
flowFile2 = session.create()
flowFile2 = session.write(flowFile2, WriteContentCallback(fraudfintransaction))
session.transfer(flowFile2, REL_SUCCESS)
session.commit()
fraud_tick = random.randint(FRAUD_TICK_MIN, FRAUD_TICK_MAX)
... View more
05-31-2023
04:38 AM
Pull Request is in!! https://github.com/apache/nifi/pull/7316 This will be fixed in future releases of NiFi. @jisaitua send me a DM with your email if you need to use the new processor right away.
... View more
05-30-2023
11:42 AM
1 Kudo
@jisaitua I am not sure how long its going to take, but i should have a github commit ready w/ a new PutBigQuery nar that you can use with 1.21. I will update again tomorrow morning.
... View more