Member since
02-01-2022
270
Posts
96
Kudos Received
59
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2178 | 06-12-2024 06:43 AM | |
3291 | 04-12-2024 06:05 AM | |
2209 | 12-07-2023 04:50 AM | |
1340 | 12-05-2023 06:22 AM | |
2261 | 11-28-2023 10:54 AM |
06-12-2023
05:24 AM
1 Kudo
@Ansh001 Just to provide an answer here. HDP is a no longer supported product and you should not be trying to build or learn on hdp sandbox. Take a look at CDP Private Cloud Base or CDP Public Cloud. As @VidyaSargur you can start conversations around becoming a customer with sales. If you are looking for self-service you can get a trial here of CDP Public Cloud: https://www.cloudera.com/campaign/try-cdp-public-cloud.html
... View more
06-09-2023
04:05 AM
@Vasu_ Can you provide screen shots of config for the processor(s) and the SSL Context Service you created? I can suggest more specific commands to build the keystore and truststore, but I will need to see what the hostname is for elastic. Additionally, if you did attempt to create a keystore/truststore from the self signed cert, be sure to share the commands you used. Commands/Code would go in a Preformatted box (top right in the full wysiwig pannel) like this
... View more
06-08-2023
06:24 AM
1 Kudo
@Vasu_ I am most definitely an expert in NIFI SSL Context Services and the various different ways to build the controller services with cacerts, public certs, and self signed certs. Here is an article i just wrote about Modern NiFi and SSL: https://community.cloudera.com/t5/Community-Articles/NIFI-SSL-in-Modern-Versions-of-NiFi/ta-p/371937 It is important to understand how to make a working SSL Context Service before trying to make custom ones. So the examples in here to use nifi's own cert's keystore and truststore, and local java cacerts as keystore/trustore, will build confidence in how to configure the controller service before trying to build custom keystores and truststores yourself. The following link has a technical example, and both solutions you can use here. Use java cacerts if your elastic endpoint is a public signed ssl cert to build a SSL Context Service Build you own custom keystore and truststore using actual cert from the endpoint and use that to build a SSL Context Service https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elasticsearch/m-p/302719
... View more
06-07-2023
08:17 AM
@Kiranq I do not believe you can replace NiFi Registry directly with Azure DevOps. You would need to keep integration with the Nifi Registry and a github integration into an Azure Devops repo.
... View more
06-07-2023
08:11 AM
1 Kudo
@SandyClouds You should really check out DataFlow. 70 jobs in one nifi, many nifis, or containerized nifi is going to be a big job to manage. Not only the setup, but the operation over time. Thats not even getting into sizing, performance, etc. These types of activities are eliminated when you deploy and operate flows in DataFlow. Here you are able to deploy multiple copies of same flow, operate them with auto scale, as well as be able to fully ci/cd the entire process to create, start, restart, etc. This latter concept is how you achieve a smooth operation of 70+ flows and never actually touch or admin nifi. Happy to demo for you if you want to take a look.
... View more
06-07-2023
07:51 AM
2 Kudos
@SandyClouds You can make nifi run as a service, then add that to startup services. This will ensure that nifi is always running after system reboots. https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#installing-as-a-service Installing as a Service Currently, installing NiFi as a service is supported only for Linux and macOS users. To install the application as a service, navigate to the installation directory in a Terminal window and execute the command bin/nifi.sh install to install the service with the default name nifi. To specify a custom name for the service, execute the command with an optional second argument that is the name of the service. For example, to install NiFi as a service with the name dataflow, use the command bin/nifi.sh install dataflow. Once installed, the service can be started and stopped using the appropriate commands, such as sudo service nifi start and sudo service nifi stop. Additionally, the running status can be checked via sudo service nifi status. Stopping or starting flows or other actions against flows would need to be done with command line or nifi rest api.
... View more
06-05-2023
06:30 AM
@drewski7 The two posts below have solutions to query the json: https://community.cloudera.com/t5/Community-Articles/Running-SQL-on-FlowFiles-using-QueryRecord-Processor-Apache/ta-p/246671 https://community.cloudera.com/t5/Support-Questions/QueryRecord-processor-issue-with-nested-JSON/td-p/338556 Here a the NiFI doc with more high level SQL info: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/latest/org.apache.nifi.processors.standard.QueryRecord/additionalDetails.html You can find other examples here in the community just search something like: queryrecord + json.
... View more
06-05-2023
06:14 AM
@Dracile If you are looking to iterate through the results inside of your upstream json object, you need QueryRecord with a Json Reader and Writer. This allows you to provide the upstream schema (reader), downstream schema (writer) and a query against the flowfile. This will unfortunately lose the original object values You can find an example here: https://github.com/cldr-steven-matison/NiFi-Templates/blob/main/QueryRecord_Sample.json You will need to modify the json object in GenerateFlowFile, then adjust the Reader/Writer, and the results query into $.VisitList[] array. Once you have this lil mini test working, take the logic to your final flow.
... View more
06-01-2023
12:40 PM
2 Kudos
@drewski7 You are looking for something like this, for "yesterday": ${now():minus(86400000):format('MM-dd-yyyy hh:mm:ss') } Then change 24 hours to 10 hours and add not reduce: ${now():plus(36000000):format('MM-dd-yyyy hh:mm:ss') } Let me know if this adapts to fit your use case!
... View more
06-01-2023
08:00 AM
3 Kudos
In this article I am going to review the required steps and processes to setup some NiFi SSL Context Services with modern versions of NiFi (1.20, 1.21, 2.0). In the past, nifi installations did not come installed with SSL enabled. If you were a NiFi admin and had to setup ssl, you know it was not an easy task, and could often times prevented the cluster from even being secured at all. Thanks to the wonderful team of innovators working on the Apache NiFi Project, I am happy to show the Easy Button works to install a fully secured Nifi. I will also show how to setup SSL Context Services for internal and external connection to https enabled endpoints. First, let's do a new install of Apache NiFi 1.21. This article is not about how to do that, so fast forward to a running NiFi, lets take a look at some important details that you will find in the nifi-app.log: Your login details: 2023-06-01 10:02:55,493 INFO [main] o.a.n.a.s.u.SingleUserLoginIdentityProvider
Generated Username [9cd754fa-9ca2-49f2-a627-53934a6876d6]
Generated Password [QQtZMerKoc9zmsGIT3X33OweIfm+nAd4]
Where you can find the NiFi UI: 2023-06-01 10:05:29,806 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs:
2023-06-01 10:05:29,806 INFO [main] org.apache.nifi.web.server.JettyServer https://127.0.0.1:8443/nifi We should be able to now login to the NiFi UI on the secured host https://[nifihost]:8443 with the provided username and password. WOW: we have a user/pass to force a login right out of the box!! Alright, now let's get started with that internal SSL Context Service. We want to create the SSL Context Service Controller Service on the root nifi canvas. This will make the ssl context service available to all our process groups. In my last article Operationalize NiFi data flows with Cloudera DataFlow , I was connecting to https://[nifihost] endpoints from within NiFi to communicate with the NiFi REST API. This process should be similar with java cacerts, or your own custom keystore(s) and truststore(s). Let's find the SSL details used to secure NIFI in conf/nifi.properties: # security properties #
nifi.sensitive.props.key=7rOfiLY584X8nNpYMdye6p2DjwfgrvW3
nifi.sensitive.props.key.protected=
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.additional.keys=
nifi.security.autoreload.enabled=false
nifi.security.autoreload.interval=10 secs
nifi.security.keystore=./conf/keystore.p12
nifi.security.keystoreType=PKCS12
nifi.security.keystorePasswd=1391fbf8ada209439bd99b95432892ca
nifi.security.keyPasswd=1391fbf8ada209439bd99b95432892ca
nifi.security.truststore=./conf/truststore.p12
nifi.security.truststoreType=PKCS12
nifi.security.truststorePasswd=0a85b23929af71e49990916bb73e1733
nifi.security.user.authorizer=single-user-authorizer
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=single-user-provider
nifi.security.user.jws.key.rotation.period=PT1H
nifi.security.ocsp.responder.url=
nifi.security.ocsp.responder.certificate= Now that we have the details we need (keystore,truststore,and passwords) we can make a new SSL Context Service like this: It is still possible to create more SSL Context Services, especially if you have self signed or custom certs attached to external endpoints you need to communicate with. To connect to most public signed certs, java's cacerts works great. This is always the first SSL Context Service I start with for connecting to public https:// endpoints. First I need to copy cacerts to my nifi conf directory: cp /usr/lib/jvm/java-11-openjdk-11.0.13.0.8-4.el8_5.x86_64/lib/security/cacerts /root/nifi-1.21.0/conf/ Now I can create a new SSL Context Service like this: The cacerts password is "changeit". In conclusion, installing a fully secured NIFI with basic user auth and SSL is now much easier than in older versions of NiFi. Additionally when setting up NiFi SSL Context Service(s) just be sure to get all the right details and they will work as expected. If you land on this article and are still struggling with setting up your own SSL Context Service, create a new community post here and give me an @steven-matison and I will be glad to help out!!
... View more
Labels: