Ask the Community

Dear Community,   After a successful installion of Cloudera Runtime 7.0.3, we tried to do a Kerberization process. (We did the same before with 5.14) Everything went fine with the Kerberos wizard, but after in the config deployment phase, the YARN NodeManagers failed to start with the following error:   Error starting NodeManagerorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:394) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:936) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1016)Caused by: java.io.IOException: Linux container executor not configured properly (error=-1) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:307) at org.apac

Hello!I'm newbe in NiFi and Groovy.I found here – "Solved: Re: Get the processor-group name in NIFI flow - Cloudera Community - 213662" (2018) – very interestion solution! Thank @mburgess !!!  I have related question.I want to build with Groovy dictionary for every proceccor where key = Proccesor ID, value = "bread crumbs" (full path to processor from root canvas – the string like this "PG1_Name" / "PG2_(Child_of_PG1)_Name" / "PG3_(Child_of_PG2)_Name" .The quesions are:How to get list of Process Groups in the root and in Each PG?How to get list of Processors in Each PG?How to organize in Groovy cycles for scan PGs?How to save / read data in/from Groovy dictionary /  NiFi cache?Thanks in advance for any help and suggestions!   

Dear Support Team,I’m reaching out to request an update to the email address linked to my Cloudera account.I have already reached to the Cloudera community but haven’t received a response yet, so I’m turning to you for assistance. Thank you in advance for your help .

Hello Folks,I’ve the issue with JoinEnrichment ProcessorThe ORIGINAL flowfile looks like below:{"ticketserialnumber" : "TSN00000012","tranheaderid" : "TH00000012","entrymethodid" : "EM6","deviceid" : "DEV22","prodid" : 1,"isbetrejectedbytrader" : true,"isexchangeticket" : false,"terdisplayid" : "TER41","createddate" : "2025-06-17 22:24:00.0","requestid" : "REQ00000012","userdisplayid" : "USR863","cartid" : "CART00000012","transactiontype" : 1}, {"ticketserialnumber" : "TSN0000001","tranheaderid" : "TH001","entrymethodid" : "EM001","deviceid" : "DEV001","prodid" : 1,"isbetrejectedbytrader" : false,"isexchangeticket" : false,"terdisplayid" : "TERM001","createddate" : "2025-06-17 22:10:00.0","requestid" : "REQ001","userdisplayid" : "USER001","cartid" : "CART001","transactiontype" : 1},And the ENRICHMENT Flowfile is looks like below:{"tsn" : "TSN00000008","uuid_schema" : "VREQ00000005","event_date" : "2025-06-17 22:26:53.610071","terminalid" : "TER25","userid" : "USR244","locationid" : "LO

We are upgrading from 2.0.0 M4 to 2.4.0 in EKS via helm charts.But encountering issue while running nifi.sh file. Nifi.sh file references nifi-env.sh which is empty. We are suspecting issue is with docker image of nifi.When we opened binary of nifi 2.4.0 which is downloaded locally from internet, I can see content inside nifi-env.sh. Same is not present inside EKS when we deploy nifi 2.4.0 docker image into EKSThis is crashing our main container. Basically its not setting PID path.

Hello I'm trying to use Nifi Hashi Corp Vault Parameter Provider with Kubernetes authentication. - Nifi versions is 2.4.0 - Hashi Corp version is 1.19.5From my Nifi pod I get to authenticate successfully using the following curl curl -s -k -X POST -H 'X-Vault-Namespace: DATA/DATAOPE/EC002I002621/' -H 'X-Vault-Request: true' -d '{"jwt": "<content of file /var/run/secrets/kubernetes.io/serviceaccount/token>", "role": "role_k8s_auth_demo_re_ns002i007839"}' https://hvault.staging.my_company/v1/auth/kubernetes_demo_re_ns002i007839/loginFrom the pod I then get to make a successful secret retrieval with the follwing curlcurl -H "X-Vault-Request: true" -H "X-Vault-Namespace: DATA/DATAOPE/EC002I002621/" -H "X-Vault-Token: <token retrieved by previous command>" https://hvault.staging.my_company/v1/secret/data/testI try to make sense of the doc and cannot get to a working configurationI setup my HashiCorp Vault Client with the following properties direct properties

Hello,Has anyone ever used formal methods or a logic language such as TlaPlus to check Apache Ranger policies?If so, could you please provide examples?

How can I send select queries to the Oracle database to be binded with the executeSql processor in Apache NiFi 1.23.2 version?

Hey everyone, I’m currently preparing for the CCA175 (Cloudera Data Engineer) exam and focusing heavily on real, hands-on scenario challenges to strengthen my understanding. So far, I’ve practiced with various ingestion and transformation pipelines, but I’m stuck on one scenario that feels very close to what the actual exam might present. Midway through my study plan, I started using Certs Matrix, which has helped me evaluate different approaches to solving Spark and Hadoop workflow problems under pressure. The scenario I’m trying to clarify is this: If you receive streaming data in inconsistent formats and need to cleanse, transform, and store it efficiently in HDFS, which approach would be most exam-accurate using Spark Structured Streaming with schema evolution, designing separate ETL pipelines for each input format, or relying on a unified schema-on-read strategy? I’d really appreciate insights from anyone who has taken CCA175 or handled similar real-world pipelines. Your guidance

According to https://hive.apache.org/general/downloads/"The Apache Hive Community has voted to declare the 3.x release line as End of Life (EOL). This means no further updates or releases will be made for this series.We urge all Hive 3.x users to upgrade to the latest versions promptly to benefit from new features and ongoing support."CDP 7.3.1 is still very much based on Hive 3.https://docs.cloudera.com/cdp-private-cloud-base/7.3.1/hive-introduction/topics/hive_whats_new_in_this_release_hive.htmlhttps://docs.cloudera.com/cdp-private-cloud-base/7.3.1/hive-introduction/topics/hive-apache-hive-3-architectural-overview.htmlWhen will Cloudera on base support Hive 4?Thanks 

Hello Cloudera Community,I'm working on a proof-of-concept for migrating our client's existing Cloudera ( CDP 7.1.4 and Impala 3.4.0 ) environment to CDP 7.1.9 with Impala 4.0. We need to validate that our workloads running on the current version will be compatible with the newer version before proceeding with the production migration.I've downloaded the trial version, but it provides CDP 7.1.4 with Impala 3.4.0. For our migration testing, we specifically need CDP 7.1.9 with Impala 4.0 to ensure accurate compatibility testing.Could you please advise on:1. How to obtain trial access to CDP 7.1.9 with Impala 4.0?2. Whether the current trial license can be used with version 7.1.9?3. Access to the CDP 7.1.9 repositories for trial purposes?This is a time-sensitive POC that will directly influence our client's decision to proceed with the CDP upgrade.Thank you for your assistance.Best regards

We were looking at storing the provenance *.gz ( lucene ) files so we could have a record of all activity through Nifi, as our IT security people are fussy about such things.It seems Nifi writes to multiple files at the same time, so this may not be possible, or perhaps there is a better way of doing this? I would welcome thoughts as to how we might do this, or maybe propose a different way to achieve the same logging outcome.Thanks in advance. 

Hi all, I recently spent a full 2 days over a rainy weekend trying to get a stand alone instance of Nifi v2.5.0 on ubuntu 22.04 trying to integrate with AD,  to use AD for Nifi authentication, but with no joy. It almost worked but couldn't get past a certain point. The problem I discovered is there doesn't seem to exist on the internet  a simple step by step process on how to set up Nifi with AD/LDAP.My set up is a linux PC called nifi1 and had an old windows 2012 R2 domain controller with LDAP running on it I decided to use.I created a domain called  testdomain.local and created a domain user called testuser in the AD User OU that I wanted to use as a nifi user. I also created a service account called svc-nifi also in the User OU that is used to log onto AD as a service.I ran a series of tests using LDAP command line queries from the linux PC whereby the service account could run LDAP queries from linux that proved it could access AD, read the AD info it needed and

Dear All,I hope you are doing well.I would like your help in understand how I can secure the deployed application.Public ApplicationsBy default, authentication for applications is enforced on all ports and users cannot create public applications. If desired, the Admin user can allow users to create public applications that can be accessed by unauthenticated users.To allow users to create public applications on an Cloudera AI Workbench:As an Admin user, turn on the feature flag in Admin > Security by selecting Allow applications to be configured with unauthenticated access.When creating a new application, select Enable Unauthenticated Access.For an existing application, in Settings select Enable Unauthenticated Access.To prevent all users from creating public applications, go to Admin > Security and deselect Allow applications to be configured with unauthenticated access. All existing public applications wi

I need to copy a MS SQL table with about 9 million records to a Clickhouse database.I did setup a QueryDatabaseTable processor to pull the table from the SQL database and a PutDatabaseRecord processor to push the records to Clickhouse db.As long as the flowfile from the QueryDatabaseTable processor has less records than the setting of Batch Size in the PutDatabaseRecord everything works fine.But when I have more records the PutDatabaseRecord creates multiple batches.When I pull 30.000 records from my source table and the Batch Size is set to 10.000 I end up with 60.000 records in the destination.Taking a look at the debug information from the PutDatabaseRecord it shows 1 insert and 3 insert batches.When I pull all 9.5 million records I end up with 650 million records in the destination.Any idea ?

NiFi Version: 2.5.0When transporting information from one site to another, I need to prepend a 0 to someones PIN as that site has different requirements. I query the data using a QueryRecord processor, like so:SELECT MsgSeqNbr, PostTime, SSN, EmployeeID, LName, FName, MName, CAST(TRIM(LCN) AS BIGINT) AS LCN, LPAD(PIN, 5, '0'), EmployeeType, ValidityCode, AgencyOwner, AgencyLocated, BadgeCreatedBy, BadgeCreatedTime, BadgeModifiedBy, BadgeModifiedTime, Clearance, Error, StatusFROM FLOWFILEHowever I get the following error from Calcite:QUERY...- Caused by:org.apache.calcite.runtime.CalciteContentException: From line 5, column 5, column X: No match found for function signature LPAD(<CHARACTER>, <NUMERIC>, <CHARACTER>).I've tried a couple different methods to get this to pass, explicitly defining the type as VARCHAR(5) for both the string and the pattern, using a static value to see if it will pass but the same result occurs. The incoming Avro data has

EnvironmentNiFi 2.7.2, 3-node cluster (3EC2 - 3ASG)ZooKeeper: 3-node ensemble (ECS)NiFi Registry (ECS)AWS Step Functions for scheduled stop/start (cost optimization)ProblemAfter an overnight shutdown/startup, the cluster becomes unstable:Nodes rapidly flap: CONNECTED → DISCONNECTED → CONNECTING (every 2-3 seconds).Error: "Have not received heartbeat from node".Critical: Disconnected node logs show NO errors/warnings - node appears healthy while coordinator reports it as disconnected.Root CauseParallel startup via Step Function Map state causes:NiFi nodes start before the ZooKeeper quorum formsAll 3 nodes start simultaneously → chaotic coordinator electionResolutionDeleted flow.xml.gz on disconnected nodes → restart → nodes rejoined successfullyProposed SolutionSequential startup with proper wait times: 1. ZK1 → 60s → ZK2 → 60s → ZK3 → 120s (quorum)2. NiFi Registry → 90s3. Node 1 → 180s → Node 2 → 120s → Node 3 → 120sSequential shutdown (reverse order)QuestionsOfficial guidance on

Has anyone got nested AD groups working with the ldap-user-group-provider? I have specified the group search scope using SUBTREE in the authoriser.xml. But granting the permission to the parent group A doesnt work for user A, and user A is a member of group B, and group B is a member of group A.  I can see both groups when assigning permissions  via Polices on the UI.Any advice would be appreciated.

Hi everyone.Just want to check as of latest CML version, has there been any support for using GPU for Spark in CML ?I have seen:1. GPU used for Spark in CDE, or2. GPU used in general Python project ( using torch, tensorflow, rapids' cudf/cuml) in CML,but can't find any for Spark in CML. If there is no such support; is there any specific reason why it cannot be done while it can be done for general Spark on k8s.Thanks for any info...

Hello Expert,I am using "ConsumeKafka_2_6 " processor to consume data in apache nifi 1.25.0.In one of the case user has configured the processors with wrong/invalid kafka details.And due to this nifi started consuming hell lot of memory and as we run nifi on k8,pod got killed with OOM and was restarting every 10 mins.We stopped each PG by PG to dig down and conclude that this processor is culprit.This issue happened only twice as of now.i.e it does not behave this all the time, may be with specific condition or so.Any suggestion/solution for this issue is much appreciated. Thanks in advance.Mahendra

HiI wanted to look into a need to transfer 1 TB files by using chunking in Nifi.Each file also has to have its 20 items of meta data associated with it remain intact so the metadata and the data both survive the breaking up of the file ( chunking ) into 1000 chunks and re-assembling the file at the destination ( de-chunking ). Also, is the meta data for the large file duplicated onto each of the file 1000 chunks or is it a sub-set of the meta data?Someone mentioned nifi passes the file chunks data through JVM memory on its way to the content repository.Can I confirm whether file chunks pass through JVM memory as they are written to the file/content repository for a large file ( or any file for that matter ?) - I was fairly sure they aren't, otherwise the JVM size ( limited by machine RAM ) on the machine would limit reading in of large file data, and that would limit large file transfer speed - is that correct?I'm trying to confirm my understanding of how Nifi handles these large files

Hi Team,We are planning to deploy the Iceberg Hive runtime JAR (iceberg-hive-runtime.jar) in our CDP Private Cloud Base 7.1.9 cluster (Cloudera Manager 7.11.3) to enable Hive and Spark to work with Iceberg tables.Iceberg Hive runtime JAR: iceberg-hive-runtime-1.3.1.jar,iceberg-spark-runtime-3.3_2.12-1.3.1.jarstep: 1Upload the JAR to HDFSCreate a directory in HDFS:hdfs dfs -mkdir -p /user/share/libUpload the JAR:hdfs dfs -put iceberg-hive-runtime.jar /user/share/lib/hdfs dfs -chmod 755 /user/share/lib/iceberg-hive-runtime.jarConfigure Hive via Cloudera ManagerNavigate to Hive → Configuration → Auxiliary JARs DirectoryAdd:hdfs:///user/share/libStep 2:Configure Spark via Cloudera ManagerNavigate to Spark → Configuration → Spark Client Advanced Configuration Snippet (Safety Valve)Add:spark.yarn.jars hdfs:///user/share/lib/iceberg-hive-runtime.jarStep 3:Restart ServicesRestart Hive, Spark, and YARN. We would like guidance and confirmation on the steps, any possible impacts, and best pr

Hi Everyone,We are evaluating whether Apache Hadoop can run inside an Istio service mesh and would like to understand if this is a supported or practical setup.Our environment:Hadoop cluster running on physical machines (bare metal)No KerberosIstio is mandatoryIstio mTLS must be enabledHadoop services (NameNode, DataNode, YARN, etc.) would need to communicate through IstioOur concerns:Hadoop uses long-lived TCP connections and custom RPC protocolsIstio mTLS intercepts and terminates connectionsUnsure if Hadoop components work reliably behind Envoy proxiesUncertain about the performance and stability impactQuestions:Has anyone successfully run Hadoop inside Istio with mTLS enabled?Is this officially supported or just theoretically possible?Are there known limitations or failure cases (HDFS, YARN, shuffle, etc.)?Any real-world experience or guidance would be greatly appreciated.Thanks!

Hello,<in a Public Cloud Environment in the Operational Database with Azure: Snapshots were failing.The HBase-Service Command Tab showes that it occurs while copying a snapshot to another directory because it could not acquire a SAS token. (*)Usually those errors are related to missing rights on Managed Identities. - But we double-checked all Azure Managed Identity IAMs and RBACs.Sometimes the SAS-Token Failure is related to Kerberos which is disabled./>What could have gone wrong? (*):Client.RangerRESTClient: ===>> RangerRESTClient.init() : Since mKeyStoreType is NULL, setting System default.[mKeyStoreType=jks] Exception in thread "main" Failed to acquire a SAS token for get-acl on / due to org.apache.hadoop.security.AccessControlException: Permission denied. at org.apache.hadoop.fs.azurebfs.services.AbfsClient.appendSASTokenToQuery(AbfsClient.java:1233)

Hello communityI was wondering if anyone has any experience with writing entries into redis with dynamic TTLs. From what I can tell, it is only possible to set the TTL via the DistributedMapCacheClientService and as such it is set 'globally' per flow. For me the ideal case would've been if it was possible to set the TTL based on flowfile attributes.Any advice would be greatly appreciated. Thanks 🙂