Q&A

Hi all, I recently spent a full 2 days over a rainy weekend trying to get a stand alone instance of Nifi v2.5.0 on ubuntu 22.04 trying to integrate with AD,  to use AD for Nifi authentication, but with no joy. It almost worked but couldn't get past a certain point. The problem I discovered is there doesn't seem to exist on the internet  a simple step by step process on how to set up Nifi with AD/LDAP.My set up is a linux PC called nifi1 and had an old windows 2012 R2 domain controller with LDAP running on it I decided to use.I created a domain called  testdomain.local and created a domain user called testuser in the AD User OU that I wanted to use as a nifi user. I also created a service account called svc-nifi also in the User OU that is used to log onto AD as a service.I ran a series of tests using LDAP command line queries from the linux PC whereby the service account could run LDAP queries from linux that proved it could access AD, read the AD info it needed and

Dear All,I hope you are doing well.I would like your help in understand how I can secure the deployed application.Public ApplicationsBy default, authentication for applications is enforced on all ports and users cannot create public applications. If desired, the Admin user can allow users to create public applications that can be accessed by unauthenticated users.To allow users to create public applications on an Cloudera AI Workbench:As an Admin user, turn on the feature flag in Admin > Security by selecting Allow applications to be configured with unauthenticated access.When creating a new application, select Enable Unauthenticated Access.For an existing application, in Settings select Enable Unauthenticated Access.To prevent all users from creating public applications, go to Admin > Security and deselect Allow applications to be configured with unauthenticated access. All existing public applications wi

I need to copy a MS SQL table with about 9 million records to a Clickhouse database.I did setup a QueryDatabaseTable processor to pull the table from the SQL database and a PutDatabaseRecord processor to push the records to Clickhouse db.As long as the flowfile from the QueryDatabaseTable processor has less records than the setting of Batch Size in the PutDatabaseRecord everything works fine.But when I have more records the PutDatabaseRecord creates multiple batches.When I pull 30.000 records from my source table and the Batch Size is set to 10.000 I end up with 60.000 records in the destination.Taking a look at the debug information from the PutDatabaseRecord it shows 1 insert and 3 insert batches.When I pull all 9.5 million records I end up with 650 million records in the destination.Any idea ?

NiFi Version: 2.5.0When transporting information from one site to another, I need to prepend a 0 to someones PIN as that site has different requirements. I query the data using a QueryRecord processor, like so:SELECT MsgSeqNbr, PostTime, SSN, EmployeeID, LName, FName, MName, CAST(TRIM(LCN) AS BIGINT) AS LCN, LPAD(PIN, 5, '0'), EmployeeType, ValidityCode, AgencyOwner, AgencyLocated, BadgeCreatedBy, BadgeCreatedTime, BadgeModifiedBy, BadgeModifiedTime, Clearance, Error, StatusFROM FLOWFILEHowever I get the following error from Calcite:QUERY...- Caused by:org.apache.calcite.runtime.CalciteContentException: From line 5, column 5, column X: No match found for function signature LPAD(<CHARACTER>, <NUMERIC>, <CHARACTER>).I've tried a couple different methods to get this to pass, explicitly defining the type as VARCHAR(5) for both the string and the pattern, using a static value to see if it will pass but the same result occurs. The incoming Avro data has

EnvironmentNiFi 2.7.2, 3-node cluster (3EC2 - 3ASG)ZooKeeper: 3-node ensemble (ECS)NiFi Registry (ECS)AWS Step Functions for scheduled stop/start (cost optimization)ProblemAfter an overnight shutdown/startup, the cluster becomes unstable:Nodes rapidly flap: CONNECTED → DISCONNECTED → CONNECTING (every 2-3 seconds).Error: "Have not received heartbeat from node".Critical: Disconnected node logs show NO errors/warnings - node appears healthy while coordinator reports it as disconnected.Root CauseParallel startup via Step Function Map state causes:NiFi nodes start before the ZooKeeper quorum formsAll 3 nodes start simultaneously → chaotic coordinator electionResolutionDeleted flow.xml.gz on disconnected nodes → restart → nodes rejoined successfullyProposed SolutionSequential startup with proper wait times: 1. ZK1 → 60s → ZK2 → 60s → ZK3 → 120s (quorum)2. NiFi Registry → 90s3. Node 1 → 180s → Node 2 → 120s → Node 3 → 120sSequential shutdown (reverse order)QuestionsOfficial guidance on

Has anyone got nested AD groups working with the ldap-user-group-provider? I have specified the group search scope using SUBTREE in the authoriser.xml. But granting the permission to the parent group A doesnt work for user A, and user A is a member of group B, and group B is a member of group A.  I can see both groups when assigning permissions  via Polices on the UI.Any advice would be appreciated.

Hi everyone.Just want to check as of latest CML version, has there been any support for using GPU for Spark in CML ?I have seen:1. GPU used for Spark in CDE, or2. GPU used in general Python project ( using torch, tensorflow, rapids' cudf/cuml) in CML,but can't find any for Spark in CML. If there is no such support; is there any specific reason why it cannot be done while it can be done for general Spark on k8s.Thanks for any info...

Hello Expert,I am using "ConsumeKafka_2_6 " processor to consume data in apache nifi 1.25.0.In one of the case user has configured the processors with wrong/invalid kafka details.And due to this nifi started consuming hell lot of memory and as we run nifi on k8,pod got killed with OOM and was restarting every 10 mins.We stopped each PG by PG to dig down and conclude that this processor is culprit.This issue happened only twice as of now.i.e it does not behave this all the time, may be with specific condition or so.Any suggestion/solution for this issue is much appreciated. Thanks in advance.Mahendra

HiI wanted to look into a need to transfer 1 TB files by using chunking in Nifi.Each file also has to have its 20 items of meta data associated with it remain intact so the metadata and the data both survive the breaking up of the file ( chunking ) into 1000 chunks and re-assembling the file at the destination ( de-chunking ). Also, is the meta data for the large file duplicated onto each of the file 1000 chunks or is it a sub-set of the meta data?Someone mentioned nifi passes the file chunks data through JVM memory on its way to the content repository.Can I confirm whether file chunks pass through JVM memory as they are written to the file/content repository for a large file ( or any file for that matter ?) - I was fairly sure they aren't, otherwise the JVM size ( limited by machine RAM ) on the machine would limit reading in of large file data, and that would limit large file transfer speed - is that correct?I'm trying to confirm my understanding of how Nifi handles these large files

Hi Team,We are planning to deploy the Iceberg Hive runtime JAR (iceberg-hive-runtime.jar) in our CDP Private Cloud Base 7.1.9 cluster (Cloudera Manager 7.11.3) to enable Hive and Spark to work with Iceberg tables.Iceberg Hive runtime JAR: iceberg-hive-runtime-1.3.1.jar,iceberg-spark-runtime-3.3_2.12-1.3.1.jarstep: 1Upload the JAR to HDFSCreate a directory in HDFS:hdfs dfs -mkdir -p /user/share/libUpload the JAR:hdfs dfs -put iceberg-hive-runtime.jar /user/share/lib/hdfs dfs -chmod 755 /user/share/lib/iceberg-hive-runtime.jarConfigure Hive via Cloudera ManagerNavigate to Hive → Configuration → Auxiliary JARs DirectoryAdd:hdfs:///user/share/libStep 2:Configure Spark via Cloudera ManagerNavigate to Spark → Configuration → Spark Client Advanced Configuration Snippet (Safety Valve)Add:spark.yarn.jars hdfs:///user/share/lib/iceberg-hive-runtime.jarStep 3:Restart ServicesRestart Hive, Spark, and YARN. We would like guidance and confirmation on the steps, any possible impacts, and best pr

Hi Everyone,We are evaluating whether Apache Hadoop can run inside an Istio service mesh and would like to understand if this is a supported or practical setup.Our environment:Hadoop cluster running on physical machines (bare metal)No KerberosIstio is mandatoryIstio mTLS must be enabledHadoop services (NameNode, DataNode, YARN, etc.) would need to communicate through IstioOur concerns:Hadoop uses long-lived TCP connections and custom RPC protocolsIstio mTLS intercepts and terminates connectionsUnsure if Hadoop components work reliably behind Envoy proxiesUncertain about the performance and stability impactQuestions:Has anyone successfully run Hadoop inside Istio with mTLS enabled?Is this officially supported or just theoretically possible?Are there known limitations or failure cases (HDFS, YARN, shuffle, etc.)?Any real-world experience or guidance would be greatly appreciated.Thanks!

Hello,<in a Public Cloud Environment in the Operational Database with Azure: Snapshots were failing.The HBase-Service Command Tab showes that it occurs while copying a snapshot to another directory because it could not acquire a SAS token. (*)Usually those errors are related to missing rights on Managed Identities. - But we double-checked all Azure Managed Identity IAMs and RBACs.Sometimes the SAS-Token Failure is related to Kerberos which is disabled./>What could have gone wrong? (*):Client.RangerRESTClient: ===>> RangerRESTClient.init() : Since mKeyStoreType is NULL, setting System default.[mKeyStoreType=jks] Exception in thread "main" Failed to acquire a SAS token for get-acl on / due to org.apache.hadoop.security.AccessControlException: Permission denied. at org.apache.hadoop.fs.azurebfs.services.AbfsClient.appendSASTokenToQuery(AbfsClient.java:1233)

Hello communityI was wondering if anyone has any experience with writing entries into redis with dynamic TTLs. From what I can tell, it is only possible to set the TTL via the DistributedMapCacheClientService and as such it is set 'globally' per flow. For me the ideal case would've been if it was possible to set the TTL based on flowfile attributes.Any advice would be greatly appreciated. Thanks 🙂 

I am new to Nifi and trying to load csv data in postgres but I am getting error:FlowFile[filename=test_data.csv]. Routing to failure.: java.lang.IllegalArgumentException: Record does not have a value for the Required column 'name'so I have checked everything and the csv file I am uploading contains the data.I have also checked solutions for issues like this but nothing worked out. I will attach liks to the screenshots it contains the configuration for PutDatabaseRecord and the query for the table I created in postgre and screenshot of the csv file. https://drive.google.com/file/d/1TCJTcNJHRslG4wEct1eZmuYz54Gp9e9o/view?usp=drive_linkhttps://drive.google.com/file/d/1Ye6pggRN4YTQVGxylNarjZh4dlX-ByQ1/view?usp=drive_linkhttps://drive.google.com/file/d/17zLRe88DSssU9DI3rrJLCHQmUgAj9FYX/view?usp=drive_linkhttps://drive.google.com/file/d/1XKt1jbZks8jE0kO4CrT3nmRR4LFcIn1e/view?usp=drive_link Note: please let me know how I can upload image here directly

I am new to Apache. I have generated files using Generate Flow File processor but when I try to view the file content it gives me UI error "Unable to communicate to NiFi" I think there is a problem with content file viewer in NiFi.I am using https and port 8443 and also when I download the file the content in the file appears all okay just that it's not working in the browser! 

Hello,We are on CDP 7.1.9 SP1 and have Hue configured to be accessible via Knox. We would like to limit the access to Hui UI to a particular user group. I did found below documentation, however, unsure if that will help incase of accessibility via Knox as it looks to be more LDAP relevant.https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/securing-hue/topics/hue-restrict-user-login.htmlPlease advise.ThanksSnm1523

I am new to nifi trying to install it but when I run command nifi.cmd start I get this:JAVA_HOME=C:\Program Files\Java\jdk-21NIFI_HOME=C:\nifi-2.7.2-bin\nifi-2.7.2but it doesn't write that "nifi has started" when I checked the log file it has the following:nar\extensions\nifi-server-nar-2.7.2.nar-unpacked\NAR-INF\bundled-dependencies\nifi-ui-2.7.2.war] Context Path [/nifi]2026-01-17 19:26:53,803 INFO [main] org.apache.nifi.web.server.JettyServer Loading WAR [C:\nifi-2.7.2-bin\nifi-2.7.2\.\work\nar\extensions\nifi-server-nar-2.7.2.nar-unpacked\NAR-INF\bundled-dependencies\nifi-web-api-2.7.2.war] Context Path [/nifi-api]2026-01-17 19:26:53,810 INFO [main] org.apache.nifi.web.server.JettyServer Loading Docs [C:\nifi-2.7.2-bin\nifi-2.7.2\docs] Context Path [/nifi-api]2026-01-17 19:26:53,813 ERROR [main] org.apache.nifi.runtime.Application Start Server failedjava.lang.IllegalStateException: Management Server Address System Property [org.apache.nifi.management.server.address] not valid [127.

Hello team & community 🙂I am looking into deploying many instances of a flow that is versioned in a git registry via the REST API. My current issue is that I would like to configure each instance of this flow dynamically, though if I version the flow with a Parameter Context attached to it, whenever I deploy instances of it they all end up attaching to a single Parameter Context. I understand this is the expected behaviour as described in the documentation. Therefore, I'd like to know if there is any recommended practice for situations such as mine. The best solution I could think of was perhaps configuring the processors in the flow to use parameters but not to actually attach a PC to the process group. This way I could deploy flows in two steps by first importing them and then creating a new parameter context & configuring/attaching it. I would greatly appreciate any advice for this use-case,Thank you.

请问，在cdp7.17 环境，执行gbk转utf8报错，如下图所示,要如何解决？  

Hi all,I've run into an interesting issue where corrupted entries with rowkeys containing a single comma (,) were inserted into hbase:meta. These rows show up in scans, but get, delete, and deleteall operations don't work on them. Example of such a row:ROW COLUMN+CELLrowkey, column=info:regioninfo, timestamp=..., value=...Attempts to remove them with the Java client and HBCK2 (fixMeta) didn't work.Has anyone dealt with this before? Is there a safe way to remove or repair such dangling/corrupted rowkeys from hbase:meta?In addition, I have opened this upstream issue reporting the problem: https://issues.apache.org/jira/browse/HBASE-29554Thanks in advance for any guidance.

Hi everyone, I’m encountering a critical issue with HBase 2.x: the RegionServer fails to connect to the Master, throwing a “GSS initiate failed” error. Environment:Master node: host117RegionServer node: host121Kerberos security is enabled To troubleshoot this, I’ve performed the following checks and fixes—all verified as successful (:white_heavy_check_mark:): Time SynchronizationClock skew across cluster nodes is only 8 seconds, well within Kerberos tolerance (typically ≤ 5 minutes).Hostname ResolutionAdded explicit entries in /etc/hosts for both host117 and host121 to ensure bidirectional hostname resolution, eliminating potential Kerberos failures due to DNS issues.Network ConnectivityConfirmed TCP connectivity to the Master’s RPC port using telnet host117 16000.Kerberos Client Configuration (/etc/krb5.conf)Verified KDC is reachable and TGS requests succeed.Confirmed support for AES256 and AES128 encryption types, matching HBase requirements.JAAS Configuration Fix

I am getting the following Checkpoint Status error and wondering if someone has an idea of how I can solve this? Brand new cluster not in use yet. Clouder Manager v7.13.1, Runtime v7.3.1 on RHEL9.Bad : The filesystem checkpoint is 12 hour(s), 41 minute(s) old. This is 1,269.03% of the configured checkpoint period of 1 hour(s). Critical threshold: 400.00%. 7,501 transactions have occurred since the last filesystem checkpoint. This is 0.75% of the configured checkpoint transaction target of 1,000,000.Role Log:11:34:31.289 AMINFOFSNamesystemRoll Edit Log from 192.168.158.211:34:31.289 AMINFOFSEditLogRolling edit logs11:34:31.289 AMINFOFSEditLogEnding log segment 95836, 9584211:34:31.290 AMINFOFSEditLogNumber of transactions: 8 Total time for transactions(ms): 2 Number of transactions batched in Syncs: 2 Number of syncs: 6 SyncTimes(ms): 5 11:34:31.290 AMINFOFSEditLogNumber of transactions: 8 Total time for transactions(ms): 2 Number of transactions batched in Syncs: 2 Number of syncs: 7 S

Hi Folks,I have 2 NameNode (HA), 16 DataNode, and 5 JournalNode. When I tried to shutdown 8 DataNode, 2 NameNode was down and not running.Please give me a solution for my case. How many journal nodes are needed to be able to handle a minimum of 8 DataNodes that are down or have a greater than 50% failure rate?Pleas Advice.....

Hi all,I'm facing an issue with HDFS 3.3.5 cluster.I have created a total of 650 files with replication set to 1 and block size set to 4MB and each file size is 4MB which should result in a subtotal of 2.5GB, but the storage utilisation is marked closer to 100% due to this.Thanks.

I have a 3 node cluster (ml1, ml2, and vortex). ml1 is the namenode, ml2 and vortex are datanodes. Last week we had a power shutdown in our school and since then i have not been able to access the files in hdfs.When i try to see the "Head of the file" i get an error: "Couldn't find datanode to read file from. Forbidden".The size of the file is correct (my understanding is that the file is still somewhere). The output of command: hdfs dfsadmin -report is as below:Configured Capacity: 2088742875136 (1.90 TB)Present Capacity: 1673267032064 (1.52 TB)DFS Remaining: 942202814464 (877.49 GB)DFS Used: 731064217600 (680.86 GB)DFS Used%: 43.69%Replicated Blocks:Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 6303Missing blocks (with replication factor 1): 6303Low redundancy blocks with highest priority to recover: 0Pending deletion blocks: 0Erasure Coded Block Groups:Low redundancy block groups: 0Block groups with corrupt internal blocks: 0Missing block groups: 0Low redu