Ask the Community

Hi,We have deployed Cloudera Flow Management through the Kubernetes Operator on a k8s cluster. What's the best practice how to add custom processor NARs in this environment?On a plain (not containerized) NiFi deployment I would put the NAR into the extensions sub folder of the NiFi deployment and it gets unpacked during startup of NiFi.How does it work with the Cloudera Kubernetes Operator version of NiFi? We have a NFS share available that could be mapped to a container's sub folder.Has anyone done this successfully?Thanks,-karsten

I can able to connect the impala using isql when I use to connect the same using pyodbc I am getting this error Error: ('H000', '[H000] [ (50404) (SQLDriverConnect)')

Hi  , I want to automate my NiFi flow deployment process.Currently, I have a NiFi template generator that works with multiple XML and configuration files where I define properties like Kafka topic names, broker IPs, and other NiFi-related settings. Using this template generator, I parse an existing sample NiFi XML template and generate a new XML file with all the required properties populated.Right now, I upload this generated XML file manually through the NiFi UI. After that, I manually create a new process group on the NiFi canvas, import the template into this group, and then start the processor group and all required controller services.I want to automate this entire flow—right from generating the template and creating the process group, to uploading the template into NiFi and starting the processors and controller services—so that no manual steps are needed in the NiFi UI.

HiWhat is the preferred method to alter a nifi processor configuration please?We want to be able to get a user to choose from one of 3 options from a drop down menu in a GUI screen. The chosen option then is passed to either a curl command to make changes to the processor via the Nifi API, or can we create a python command to then alter a nifi processor configuration possibly by curl, but maybe there is another way?Curl looks a safer option, but python might be better? Not sure....?Thanks in advance. 

Hi everyone, I hope you’re doing well. I am working on a dataflow in Apache NiFi 1.18, and I need to retrieve the queue size information (flowfile count and content size) directly within NiFi itself, not via an external script or Postman. I know that the NiFi REST API provides this data, and I can access it successfully using external tools. However, my goal is to access queue metrics from inside NiFi, for example through processors like InvokeScriptedProcessor, QueryNiFiReportingTask, or any other built-in mechanism, without sending an external REST API request from outside NiFi. Is there a recommended approach, processor, or reporting task that allows NiFi to read its own queue sizes internally? If not, what would be the best practice to achieve this? Any guidance or examples would be greatly appreciated. Thank you in advance!

Hi everyone,I installed Spark cluster standalone. I want to config send log job to minio. I configured spark-default.conf file as bellow.  spark.eventLog.enabled=false spark.eventLog.dir=s3a://spark-logs/ spark.history.fs.logDirectory=s3a://spark-logs/ spark.hadoop.fs.s3a.endpoint=http://192.168.182.131:9000 spark.hadoop.fs.s3a.access.key=admin spark.hadoop.fs.s3a.secret.key=admin12345 spark.hadoop.fs.s3a.path.style.access=true"  When I submit job spark-submit --master spark://spark-master-svc:7077 --conf spark.jars.ivy=/tmp/.ivy2 pi.py 1. I got the error:5/10/26 12:13:29 INFO SparkContext: Successfully stopped SparkContextTraceback (most recent call last): File "/opt/bitnami/spark/examples/src/main/python/pi.py", line 32, in <module> .getOrCreate() ^^^^^^^^^^^^^ File "/opt/bitnami/spark/python/lib/pyspark.zip/pyspark/sql/session.py", line 497, in getOrCreate File "/opt/bitnami/spark/python/lib/pyspark.zip/pyspark/context.py", line 51

I’m currently migrating from Apache NiFi 1.21 to NiFi 2.6.0. In my existing flow, I was using the InvokeGRPC processor, but I noticed that this processor has been removed in NiFi 2.x versions.I have a few questions:What is the exact reason for deprecating and removing InvokeGRPC in NiFi 2.x?Was it due to architectural changes, performance issues, or security concerns?If we take a fork of the NiFi 1.21 gRPC NAR and use it in NiFi 2.6.0, will that work?Are there any known performance risks with this approach?Could this introduce security vulnerabilities in the current or future releases?Is there an alternative recommended approach for gRPC integration in NiFi 2.x?For example, should we use ExecuteScript, custom processors, or any official replacement?Any insights or official guidance would be greatly appreciated. Thanks in advance!

I am setting up a 3-node Apache Kafka cluster using KRaft mode (3.9.0) with SCRAM-SHA-256 for inter-broker communication on the SASL_PLAINTEXT listener. The cluster is unable to form a quorum due to persistent SASL authentication failures.All configuration files and passwords have been synchronized. The exact error points to a handshake failure, but the settings appear correct.Environment and Node Details Kafka Version: 3.9.0 (KRaft mode)Security Protocol: SASL_PLAINTEXT (SCRAM-SHA-256)JAAS File Location: /opt/kafka/config/kafka_server_jaas.confNodes:Node 1: 172.20.24.155 (node.id=1)Node 2: 172.20.24.156 (node.id=2)Node 3: 172.20.24.157 (node.id=3)Errors The primary error is a SASL handshake failure, leading the client (connecting broker) to prematurely send an API request. This confirms that the broker is attempting to authenticate but is immediately rejected by the remote server.NODE 1 Logs (Example of persistent failures after configuration changes)Nov 13 13

Hello,During a security scan, two issues have been discovered in my Cloudera cluster (Cloudera Manager version 7.13.1, Cloudera Runtime 7.3.1).Zookeeper issue (Zookeeper version 3.8.1.7.3.1.0-197):On the nodes where zookeeper runs, I have the bellow issue:HTTP TRACE / TRACK Methods Allowed - Debugging functions are enabled on the remote web server - port 7000Mapreduce HSTS header:HSTS Missing From HTTPS Server / port 13562 Can you please assist on how to mitigate these issues? Thank you! 

Sqoop command is failing for Oracle database on CDP 7.1.9 cluster and same command works on different cluster of same versionThe only difference between working and non-working is storage, it might not be the main issue.sqoop command: sqoop import -libjars ./ojdbc8.jar -Doracle.jdbc.defaultNChar=true -Doracle.jdbc.OracleConnection.CONNECTION_PROPERTY_SET_ROW_PREFETCH_SIZE=10 -Doracle.jdbc.timezoneAsRegion=false -Doracle.sessionTimeZone=Asia/Kolkata -Doracle.jdbc.NLS_LANG=AMERICAN_AMERICA.AL32UTF8 -Doracle.jdbc.timezoneAsRegion=false --connect jdbc:oracle:thin:@hostname:1522:servicename --password $password --map-column-java CHAR_COL=String,DATE_COL=String,FLOAT_COL=Double,TIMESTAMP_COL=String,VARCHAR_COL=String,VARCHAR2_COL=String,NUMPERPS_COL=String,LONG_COL=String --username $username --as-avrodatafile --split-by FLOAT_COL --mapreduce-job-name SQOOP_SOURCE --columns CHAR_COL,DATE_COL,FLOAT_COL,TIMESTAMP_COL,VARCHAR_COL,VARCHAR2_COL,NUMPERPS_COL,LONG_COL --delete-target-dir --tab

I am facing below error all jobs are failing because of this.this is the error give guidance why it occurs because of this error all jobs failed also please share what other relevant information needed.Data integration service failed to create DIM instance because of the following error:jave.lang runtime exception java.IOException[error while authentication with endpoint. https://sitlxdvdlap097/kms/v1?op=GETDELEGATIONTOKEN&doAs=k164prda&renewer=K164PRDA%40SAIBSIT.COM]AUTHENTICATION FAILED, URL https://sitlxdvdlap097.saibsit.com:9494/kms/v1/?op==GETDELEGATIONTOKEN&doAs=k164prda&renewer==K164PRDA%40SAIBSIT.COM&user.name=K164PRDA, status=403 message: null ]

I am currently pursuing the CLOUDERA AI Technical Professional Accreditation (CAITP) certification. I have completed the first two courses successfully; however, I am unable to proceed to the third course because there is no “Launch” button displayed.Additionally, even though the first two courses show as completed individually, the overall certification progress is not updated and does not reflect the completed percentage.I would like to continue the remaining courses, but there is no option or path to proceed further.For reference, I have attached screenshots showing:Completed status of the first two coursesMissing launch button for the third courseLack of progress update on the overall CAITP certificationFor comparison, I have already completed the CLOUDERA AI Sales Professional Accreditation (CAISP) certification successfully without encountering this issue.Can you please assist me in resolving this and enabling access to the remaining course

Hi everyone,Can i have something for Cloudera CDP 7.3.1 sandbox and practice env for deploying and upgrade. and other tasks like adding data services.Please do let me know 

Hello, All!I am sorry if the questions seems stupid, but I am a newbie. Is it possible to install cloudera on premises free trial version for on a single laptop for learning purposes?Or i need a cluster of laptops to be able to run cloudera and hadoop services? 

Hi, I am looking how to handle cases when json format may vary between different client responses. On response{"data": [{"ID": "123","Name": "Queen"},{"ID": "456","Name": "King"}]}we apply following jolt transform:[{"operation": "shift","spec": {"data": {"*": {"ID": {"456": {"@(2,Name)": "Name"}}}}}}]but some times response is coming without leading "{"data":and our transformation fails. So, generally, how can we query json with JOLT if we know for instance that Name and ID are next to each other somewhere in the structure?  

Hi I am using the CDP 5 days trail versionon the CDE I tried to add the Resource, for adding the resource we need the Virtual cluster, for creating the VC we need to enable the serviceon CDE, I tried to enable the service, on there couldn't able to select the environment, by default environment is there on the dropdown, but couldn't able to select that,i checked on the management console as well, the status is running and good health.for adding the resource it is asking to check with admin, i don't know who is the admin to connect,how to fix this to create the CDE VC. 

Hi!When I try:https://repository.cloudera.com/repository/cloudera-repos/org/eclipse/collections/eclipse-collections-api/maven-metadata.xmlI get a 500 error.curl -I "https://repository.cloudera.com/repository/cloudera-repos/org/eclipse/collections/eclipse-collections-api/maven-metadata.xml"HTTP/2 500date: Wed, 19 Nov 2025 16:42:49 GMTcontent-type: text/htmlcontent-length: 1596server: nginxx-content-type-options: nosniffcontent-security-policy: sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigationx-xss-protection: 1; mode=blockx-frame-options: DENYcache-control: no-cache, no-store, max-age=0, must-revalidate, post-check=0, pre-check=0pragma: no-cacheexpires: 0

Current Issue:Recently, we have encountered a significant slowdown during NiFi cluster restarts. The startup process takes approximately 10 to 15 minutes, and the `nifi-app.log` indicates that NiFi is spending most of this time restoring components. For example, we see many log entries like: 2025-11-15 15:13:51,911 INFO [main] o.a.n.f.s.StandardVersionedComponentSynchronizer Added Connection[ID=70a5878d-0199-1000-ffff-ffff9a65a5f2, Source ID=7097063b-0199-1000-0000-0000565ddd06, Dest ID=7096bae9-0199-1000-ffff-ffffbae5c592] to StandardProcessGroup[identifier=70967de1-0199-1000-0000-00001dde7b99,name=Filter empty files]2025-11-15 15:13:51,913 INFO [main] o.a.n.f.s.StandardVersionedComponentSynchronizer Added Connection[ID=7097150c-0199-1000-ffff-ffffd3ea7bb1, Source ID=7096aedf-0199-1000-0000-000061416339, Dest ID=7097063b-0199-1000-0000-0000565ddd06] to StandardProcessGroup[identifier=70967de1-0199-1000-0000-00001dde7b99,name=Filter empty files]2025-11-15 15:13:51,914 INFO [main]

I am trying to read a json field from mySQL using executeSQLRecord and insert into snowflake VARIANT field type using PutDataBaseRecord in NiFi 2.6.0.its giving me this error "03:20:05 UTCERROR1695731e-9b6f-3cb8-5a9a-98ab38589597PutDatabaseRecord[id=1695731e-9b6f-3cb8-5a9a-98ab38589597] Failed to put Records to database for FlowFile[filename=c6087904-71a5-479c-9301-2c99883f5f59]. Routing to failure.: net.snowflake.client.jdbc.SnowflakeSQLException: SQL compilation error:Expression type does not match column data type, expecting VARIANT but got VARCHAR(2) for column JSON_STORE"how do I fix this? 

In Cluster mode, on Linux REDHAT 8 , (version NIFI 1.26), we have an error when we are trying to launch NIFI in service mode with command: systemctl start nifi.service.user to execute systemctl : rootfirst script executed by systemctl (start) : /etc/rc.d/init.d/nifi  In this script (line 28) will have to execute command : /install_directory/nifi/bin/nifi.shScript nifi.sh can be launched by everyone-rwxrwxr-x. 1 user usrgroup vy9 17308 Jul 24 10:49 nifi.sh In the journalctl log we have this trace:Oct 22 10:53:37 Serveur_Name nifi[110953]: /etc/rc.d/init.d/nifi: line 28: /install_directory/nifi/bin/nifi.sh: Permission deniedOct 22 10:53:37 Serveur_Name systemd[1]: nifi.service: Control process exited, code=exited status=126Oct 22 10:53:37 Serveur_Name systemd[1]: nifi.service: Failed with result 'exit-code'.I don't know which is wrong ...   

Hello,We are using hdfs 2.7.3, and we have set the storage policy of one folder to cold (Archive). We have restarted dediated services on the datanodes (the namenode services weren't asked to be restarted). I know that we have to launch a mover process to move the existing blocks which violate the storage policy. However, it looks like the new data is not automatically write into the cold storage and still need the mover. How does it work? Did I miss anything? Is there a way to make the block placement automatic to the cold storage location?*What I'm sure: the storage policy is applied (checked from getStoragePolicy and fsck command)Thank you. 

We want to move from NiFi 1.28.1 -> NiFi 2.4. We have done most of the prerequisites like using the json flow file not xml but are confused with the deprecation of the the ./tls-toolkit.sh. How are NiFi 2.0+ clusters secured? We are using helm charts. 

Hello dear community,I haven’t been able to find a solution for the following setup:I have Apache NiFi 2.5.0 and NiFi Registry 2.5.0 installed on the same EC2 instance (Linux 2023). Both NiFi and NiFi Registry are configured with SSL and served through an Application Load Balancer (ALB).External access is only available via the ALB: https://alb...com.Internally, the services can be reached at:NiFi: https://nifi_ec2server.com:8443/nifiNiFi Registry: https://nifi_ec2server.com:18443/nifi-registryAuthentication is handled via Active Directory (LDAP).Accessing NiFi through https://alb.../nifi works fine — I can log in as an LDAP user. The same applies to NiFi Registry under https://alb.../nifi-registry.The problem:As an LDAP user, I cannot access the NiFi Registry from within NiFi, and no buckets are displayed.The error message is attached below.[NiFi Registry Web Server-50] o.a.n.r.w.s.a.IdentityFilter Attempting to extract user credentials using X509IdentityProvider[NiFi Registry Web Ser

I’m currently working on a requirement to create a dashboard in Cloudera based on a data flow that is ingested via OCR (Optical Character Recognition).I’m completely new to Cloudera and still exploring its ecosystem. So far, I’ve found the Cloudera AI Technical Professional Accreditation, which seems helpful, but I’d like to know if there are any other specific learning paths, courses, or documentation that could guide me through this use case.My goal is to understand the end-to-end flow from OCR data ingestion → data processing → dashboard visualization within Cloudera.Any suggestions, resources, or recommended starting points would be greatly appreciated.

Hello Everyone,                          As you know that , it is recommended to use Parameter rather than variables in Nifi as it is gonna get deprecated soon.                         I have two process groups PG1 and PG1A. PG1A is the child process group to PG1.                        I have ParameterContext PC1 which has database and other common application configuration parameters and it is assigned to PG1.                        I would like to create new ParameterContext PC1A which has its own parameters specific to the process group PG1A and it should also inherit the common application parameters from PC1. How it can be achieved? Since we can't assign two different Parameter contexts(PC1 & PC1A) to one process group (PG1A)