Cloudera Community
steven-matison
user rank
user rank
Guru

Re: Unable to register environment

by Guru Guru in Support Questions
‎06-13-2023 06:55 AM
1 Kudo
‎06-13-2023 06:55 AM
1 Kudo
@wert_1311 Your error indicates that two of your roles are missing or incomplete.   1. Data Access Role (arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role) is not set up correctly. Please follow the official documentation on required policies for Data Access Role. Missing policies (chunked): arn:aws:iam::8859X2XX911XX:role/Cloudera-datalake-admin-role:s3:AbortMultipartUpload:arn:aws:s3:::cdp-my-bucket/hive_replica_functions_dir/* 2. Data Access Role (arn:aws:iam::8859X2XX911XX:role/Cloudera-ranger-audit-role) is not set up correctly. Please follow the official documentation on required policies for Data Access Role. Missing policies (chunked): arn:aws:iam::8859X2XX911XX:role/Cloudera-ranger-audit-role:s3:PutObject:arn:aws:s3:::cdp-my-bucket/ranger/audit/* Go back to the quickstart and docs, and make sure completd all the setups.  Here is a link with more about the credentials   https://docs.cloudera.com/cdp-public-cloud/cloud/requirements-aws/topics/mc-aws-req-credential.html   There are steps in the page above describing the 2 roles which have conflicts in your error.  The error indicates, in the end of each message, which policies are missing.     ... View more

Re: Looking for some help on how to use QueryElast...

by Guru Guru in Support Questions
‎06-09-2023 04:05 AM
‎06-09-2023 04:05 AM
@Vasu_   Can you provide screen shots of config for the processor(s) and the SSL Context Service you created? I can suggest more specific commands to build the keystore and truststore, but I will need to see what the hostname is for elastic.   Additionally, if you did attempt to create a keystore/truststore from the self signed cert,  be sure to share the commands you used.   Commands/Code would go in a Preformatted box (top right in the full wysiwig pannel) like this ... View more

Re: Looking for some help on how to use QueryElast...

by Guru Guru in Support Questions
‎06-08-2023 06:24 AM
1 Kudo
‎06-08-2023 06:24 AM
1 Kudo
@Vasu_  I am most definitely an expert in NIFI SSL Context Services and the various different ways to build the controller services with cacerts, public certs, and self signed certs.   Here is an article i just wrote about Modern NiFi and SSL: https://community.cloudera.com/t5/Community-Articles/NIFI-SSL-in-Modern-Versions-of-NiFi/ta-p/371937   It is important to understand how to make a working SSL Context Service before trying to make custom ones.  So the examples in here to use nifi's own cert's keystore and truststore, and local java cacerts as keystore/trustore, will build confidence in how to configure the controller service before trying to build custom keystores and truststores yourself.   The following link has a technical example, and both solutions you can use here.   Use java cacerts if your elastic endpoint is a public signed ssl cert to build a SSL Context Service Build you own custom keystore and truststore using actual cert from the endpoint and use that to build a SSL Context Service https://community.cloudera.com/t5/Support-Questions/Configure-StandardSSLContextService-for-Elasticsearch/m-p/302719   ... View more

Re: Store Nifi Workflows in Azure Devops

by Guru Guru in Support Questions
‎06-07-2023 08:17 AM
‎06-07-2023 08:17 AM
@Kiranq I do not believe you can replace NiFi Registry directly with Azure DevOps.  You would need to keep integration with the Nifi Registry and a github integration into an Azure Devops repo.  ... View more

Re: Nifi cluster or standalone, Nifi Docker or wit...

by Guru Guru in Support Questions
‎06-07-2023 08:11 AM
1 Kudo
‎06-07-2023 08:11 AM
1 Kudo
@SandyClouds You should really check out DataFlow.     70 jobs in one nifi, many nifis, or containerized nifi is going to be a big job to manage.  Not only the setup, but the operation over time.   Thats not even getting into sizing, performance, etc.   These types of activities are eliminated when you deploy and operate flows in DataFlow.   Here you are able to deploy multiple copies of same flow, operate them with auto scale, as well as be able to fully ci/cd the entire process to create, start, restart, etc.   This latter concept is how you achieve a smooth operation of 70+ flows and never actually touch or admin nifi.   Happy to demo for you if you want to take a look. ... View more

Re: start nifi automatically when system boots

by Guru Guru in Support Questions
‎06-07-2023 07:51 AM
3 Kudos
‎06-07-2023 07:51 AM
3 Kudos
@SandyClouds You can make nifi run as a service, then add that to startup services.  This will ensure that nifi is always running after system reboots.      https://nifi.apache.org/docs/nifi-docs/html/getting-started.html#installing-as-a-service Installing as a Service Currently, installing NiFi as a service is supported only for Linux and macOS users. To install the application as a service, navigate to the installation directory in a Terminal window and execute the command bin/nifi.sh install to install the service with the default name nifi. To specify a custom name for the service, execute the command with an optional second argument that is the name of the service. For example, to install NiFi as a service with the name dataflow, use the command bin/nifi.sh install dataflow. Once installed, the service can be started and stopped using the appropriate commands, such as sudo service nifi start and sudo service nifi stop. Additionally, the running status can be checked via sudo service nifi status.   Stopping or starting flows or other actions against flows would need to be done with command line or nifi rest api.     ... View more

Re: How to split particular data within json using...

by Guru Guru in Support Questions
‎06-05-2023 06:14 AM
‎06-05-2023 06:14 AM
@Dracile If you are looking to iterate through the results inside of your upstream json object,  you need QueryRecord with a Json Reader and Writer.  This allows you to provide the upstream schema (reader), downstream schema (writer) and a query against the flowfile.     This will unfortunately lose the original object values    You can find an example here: https://github.com/cldr-steven-matison/NiFi-Templates/blob/main/QueryRecord_Sample.json     You will need to modify the json object in GenerateFlowFile, then adjust the Reader/Writer, and the results query into $.VisitList[] array.   Once you have this lil mini test working, take the logic to your final flow.   ... View more

Re: NiFi - Convert Current time to GMT+10:00

by Guru Guru in Support Questions
‎06-01-2023 12:40 PM
2 Kudos
‎06-01-2023 12:40 PM
2 Kudos
@drewski7   You are looking for something like this,  for "yesterday":   ${now():minus(86400000):format('MM-dd-yyyy hh:mm:ss') } Then change 24 hours to 10 hours and add not reduce:   ${now():plus(36000000):format('MM-dd-yyyy hh:mm:ss') }   Let me know if this adapts to fit your use case! ... View more

NIFI SSL in Modern Versions of NiFi

by Guru Guru in Community Articles
‎06-01-2023 08:00 AM
3 Kudos
‎06-01-2023 08:00 AM
3 Kudos
In this article I am going to review the required steps and processes to setup some NiFi SSL Context Services with modern versions of NiFi (1.20, 1.21, 2.0).    In the past, nifi installations did not come installed with SSL enabled.   If you were a NiFi admin and had to setup ssl, you know it was not an easy task, and could often times prevented the cluster from even being secured at all.   Thanks to the wonderful team of innovators working on the Apache NiFi Project, I am happy to show the Easy Button works to install a fully secured Nifi.  I will also show how to setup SSL Context Services for internal and external connection to https enabled endpoints.   First, let's do a new install of Apache NiFi 1.21.   This article is not about how to do that, so fast forward to a running NiFi,  lets take a look at some important details that you will find in the nifi-app.log:   Your login details: 2023-06-01 10:02:55,493 INFO [main] o.a.n.a.s.u.SingleUserLoginIdentityProvider Generated Username [9cd754fa-9ca2-49f2-a627-53934a6876d6] Generated Password [QQtZMerKoc9zmsGIT3X33OweIfm+nAd4] Where you can find the NiFi UI: 2023-06-01 10:05:29,806 INFO [main] org.apache.nifi.web.server.JettyServer NiFi has started. The UI is available at the following URLs: 2023-06-01 10:05:29,806 INFO [main] org.apache.nifi.web.server.JettyServer https://127.0.0.1:8443/nifi   We should be able to now login to the NiFi UI on the secured host https://[nifihost]:8443 with the provided username and password.  WOW: we have a user/pass to force a login right out of the box!!   Alright, now let's get started with that internal SSL Context Service.  We want to create the SSL Context Service Controller Service on the root nifi canvas.  This will make the ssl context service available to all our process groups.  In my last article Operationalize NiFi data flows with Cloudera DataFlow ,  I was connecting to https://[nifihost] endpoints from within NiFi to communicate with the NiFi REST API.     This process should be similar with java cacerts, or your own custom keystore(s) and truststore(s).   Let's find the SSL details used to secure NIFI in conf/nifi.properties: # security properties # nifi.sensitive.props.key=7rOfiLY584X8nNpYMdye6p2DjwfgrvW3 nifi.sensitive.props.key.protected= nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256 nifi.sensitive.props.additional.keys= nifi.security.autoreload.enabled=false nifi.security.autoreload.interval=10 secs nifi.security.keystore=./conf/keystore.p12 nifi.security.keystoreType=PKCS12 nifi.security.keystorePasswd=1391fbf8ada209439bd99b95432892ca nifi.security.keyPasswd=1391fbf8ada209439bd99b95432892ca nifi.security.truststore=./conf/truststore.p12 nifi.security.truststoreType=PKCS12 nifi.security.truststorePasswd=0a85b23929af71e49990916bb73e1733 nifi.security.user.authorizer=single-user-authorizer nifi.security.allow.anonymous.authentication=false nifi.security.user.login.identity.provider=single-user-provider nifi.security.user.jws.key.rotation.period=PT1H nifi.security.ocsp.responder.url= nifi.security.ocsp.responder.certificate= Now that we have the details we need (keystore,truststore,and passwords) we can make a new SSL Context Service like this:      It is still possible to create more SSL Context Services, especially if you have self signed or custom certs attached to external endpoints you need to communicate with.  To connect to most public signed certs, java's cacerts works great.   This is always the first SSL Context Service I start with for connecting to public https:// endpoints.      First I need to copy cacerts to my nifi conf directory: cp /usr/lib/jvm/java-11-openjdk-11.0.13.0.8-4.el8_5.x86_64/lib/security/cacerts /root/nifi-1.21.0/conf/ Now I can create a new SSL Context Service like this:   The cacerts password is "changeit".   In conclusion, installing a fully secured NIFI with basic user auth and SSL is now much easier than in older versions of NiFi.    Additionally when setting up NiFi SSL Context Service(s) just be sure to get all the right details and they will work as expected.    If you land on this article and are still struggling with setting up your own SSL Context Service, create a new community post here and give me an @steven-matison and I will be glad to help out!!     ... View more
Labels:

Re: NiFi Distributed Map Cache Persisting

by Guru Guru in Support Questions
‎06-01-2023 06:31 AM
1 Kudo
‎06-01-2023 06:31 AM
1 Kudo
@drewski7   I think the solution you are looking for is to use one of the alternative data stores that lives outside of NiFi with the map cache.  The options are Redis, Hbase, Cassandra, Couchbase:     They will give you greater control and they are preferred for production and large volumes. ... View more
Contact Me
Online
Offline
Last Visited
‎05-15-2026 09:38 AM
Community Statistics
Member Since ‎02-01-2022 01:27 PM
Last Visited ‎05-15-2026 09:38 AM
Posts 281
Kudos received 102