Ask the Community

I am new to Nifi and trying to load csv data in postgres but I am getting error:FlowFile[filename=test_data.csv]. Routing to failure.: java.lang.IllegalArgumentException: Record does not have a value for the Required column 'name'so I have checked everything and the csv file I am uploading contains the data.I have also checked solutions for issues like this but nothing worked out. I will attach liks to the screenshots it contains the configuration for PutDatabaseRecord and the query for the table I created in postgre and screenshot of the csv file. https://drive.google.com/file/d/1TCJTcNJHRslG4wEct1eZmuYz54Gp9e9o/view?usp=drive_linkhttps://drive.google.com/file/d/1Ye6pggRN4YTQVGxylNarjZh4dlX-ByQ1/view?usp=drive_linkhttps://drive.google.com/file/d/17zLRe88DSssU9DI3rrJLCHQmUgAj9FYX/view?usp=drive_linkhttps://drive.google.com/file/d/1XKt1jbZks8jE0kO4CrT3nmRR4LFcIn1e/view?usp=drive_link Note: please let me know how I can upload image here directly

I am new to Apache. I have generated files using Generate Flow File processor but when I try to view the file content it gives me UI error "Unable to communicate to NiFi" I think there is a problem with content file viewer in NiFi.I am using https and port 8443 and also when I download the file the content in the file appears all okay just that it's not working in the browser! 

Hello,We are on CDP 7.1.9 SP1 and have Hue configured to be accessible via Knox. We would like to limit the access to Hui UI to a particular user group. I did found below documentation, however, unsure if that will help incase of accessibility via Knox as it looks to be more LDAP relevant.https://docs.cloudera.com/cdp-private-cloud-base/7.1.9/securing-hue/topics/hue-restrict-user-login.htmlPlease advise.ThanksSnm1523

I am new to nifi trying to install it but when I run command nifi.cmd start I get this:JAVA_HOME=C:\Program Files\Java\jdk-21NIFI_HOME=C:\nifi-2.7.2-bin\nifi-2.7.2but it doesn't write that "nifi has started" when I checked the log file it has the following:nar\extensions\nifi-server-nar-2.7.2.nar-unpacked\NAR-INF\bundled-dependencies\nifi-ui-2.7.2.war] Context Path [/nifi]2026-01-17 19:26:53,803 INFO [main] org.apache.nifi.web.server.JettyServer Loading WAR [C:\nifi-2.7.2-bin\nifi-2.7.2\.\work\nar\extensions\nifi-server-nar-2.7.2.nar-unpacked\NAR-INF\bundled-dependencies\nifi-web-api-2.7.2.war] Context Path [/nifi-api]2026-01-17 19:26:53,810 INFO [main] org.apache.nifi.web.server.JettyServer Loading Docs [C:\nifi-2.7.2-bin\nifi-2.7.2\docs] Context Path [/nifi-api]2026-01-17 19:26:53,813 ERROR [main] org.apache.nifi.runtime.Application Start Server failedjava.lang.IllegalStateException: Management Server Address System Property [org.apache.nifi.management.server.address] not valid [127.

Hello team & community 🙂I am looking into deploying many instances of a flow that is versioned in a git registry via the REST API. My current issue is that I would like to configure each instance of this flow dynamically, though if I version the flow with a Parameter Context attached to it, whenever I deploy instances of it they all end up attaching to a single Parameter Context. I understand this is the expected behaviour as described in the documentation. Therefore, I'd like to know if there is any recommended practice for situations such as mine. The best solution I could think of was perhaps configuring the processors in the flow to use parameters but not to actually attach a PC to the process group. This way I could deploy flows in two steps by first importing them and then creating a new parameter context & configuring/attaching it. I would greatly appreciate any advice for this use-case,Thank you.

请问，在cdp7.17 环境，执行gbk转utf8报错，如下图所示,要如何解决？  

Hi all,I've run into an interesting issue where corrupted entries with rowkeys containing a single comma (,) were inserted into hbase:meta. These rows show up in scans, but get, delete, and deleteall operations don't work on them. Example of such a row:ROW COLUMN+CELLrowkey, column=info:regioninfo, timestamp=..., value=...Attempts to remove them with the Java client and HBCK2 (fixMeta) didn't work.Has anyone dealt with this before? Is there a safe way to remove or repair such dangling/corrupted rowkeys from hbase:meta?In addition, I have opened this upstream issue reporting the problem: https://issues.apache.org/jira/browse/HBASE-29554Thanks in advance for any guidance.

Hi everyone, I’m encountering a critical issue with HBase 2.x: the RegionServer fails to connect to the Master, throwing a “GSS initiate failed” error. Environment:Master node: host117RegionServer node: host121Kerberos security is enabled To troubleshoot this, I’ve performed the following checks and fixes—all verified as successful (:white_heavy_check_mark:): Time SynchronizationClock skew across cluster nodes is only 8 seconds, well within Kerberos tolerance (typically ≤ 5 minutes).Hostname ResolutionAdded explicit entries in /etc/hosts for both host117 and host121 to ensure bidirectional hostname resolution, eliminating potential Kerberos failures due to DNS issues.Network ConnectivityConfirmed TCP connectivity to the Master’s RPC port using telnet host117 16000.Kerberos Client Configuration (/etc/krb5.conf)Verified KDC is reachable and TGS requests succeed.Confirmed support for AES256 and AES128 encryption types, matching HBase requirements.JAAS Configuration Fix

I am getting the following Checkpoint Status error and wondering if someone has an idea of how I can solve this? Brand new cluster not in use yet. Clouder Manager v7.13.1, Runtime v7.3.1 on RHEL9.Bad : The filesystem checkpoint is 12 hour(s), 41 minute(s) old. This is 1,269.03% of the configured checkpoint period of 1 hour(s). Critical threshold: 400.00%. 7,501 transactions have occurred since the last filesystem checkpoint. This is 0.75% of the configured checkpoint transaction target of 1,000,000.Role Log:11:34:31.289 AMINFOFSNamesystemRoll Edit Log from 192.168.158.211:34:31.289 AMINFOFSEditLogRolling edit logs11:34:31.289 AMINFOFSEditLogEnding log segment 95836, 9584211:34:31.290 AMINFOFSEditLogNumber of transactions: 8 Total time for transactions(ms): 2 Number of transactions batched in Syncs: 2 Number of syncs: 6 SyncTimes(ms): 5 11:34:31.290 AMINFOFSEditLogNumber of transactions: 8 Total time for transactions(ms): 2 Number of transactions batched in Syncs: 2 Number of syncs: 7 S

Hi Folks,I have 2 NameNode (HA), 16 DataNode, and 5 JournalNode. When I tried to shutdown 8 DataNode, 2 NameNode was down and not running.Please give me a solution for my case. How many journal nodes are needed to be able to handle a minimum of 8 DataNodes that are down or have a greater than 50% failure rate?Pleas Advice.....

Hi all,I'm facing an issue with HDFS 3.3.5 cluster.I have created a total of 650 files with replication set to 1 and block size set to 4MB and each file size is 4MB which should result in a subtotal of 2.5GB, but the storage utilisation is marked closer to 100% due to this.Thanks.

I have a 3 node cluster (ml1, ml2, and vortex). ml1 is the namenode, ml2 and vortex are datanodes. Last week we had a power shutdown in our school and since then i have not been able to access the files in hdfs.When i try to see the "Head of the file" i get an error: "Couldn't find datanode to read file from. Forbidden".The size of the file is correct (my understanding is that the file is still somewhere). The output of command: hdfs dfsadmin -report is as below:Configured Capacity: 2088742875136 (1.90 TB)Present Capacity: 1673267032064 (1.52 TB)DFS Remaining: 942202814464 (877.49 GB)DFS Used: 731064217600 (680.86 GB)DFS Used%: 43.69%Replicated Blocks:Under replicated blocks: 0Blocks with corrupt replicas: 0Missing blocks: 6303Missing blocks (with replication factor 1): 6303Low redundancy blocks with highest priority to recover: 0Pending deletion blocks: 0Erasure Coded Block Groups:Low redundancy block groups: 0Block groups with corrupt internal blocks: 0Missing block groups: 0Low redu

Hello,I'm trying to insert data into a hive table configured with hdfs router based federation as table locationLocation: | hdfs://router_host:8888/router/router.db/router_test_tableCluster is kebrerized and all components including Hive and RBF are working as expected except this specific use case.The hive table insert job is failing with kerberos error when using RBF as the table locationCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: DestHost:destPort router_host:8888 , LocalHost:localPort datanode_host. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:639)at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:563)... 17 moreCaused by: java.io.IOException: DestHost:destPort router_host:8888 , LocalHost:localPort datanode_ho

Hi everyone,I'm using Hadoop 3.1.1 and have encountered an issue: after running the DataNode for a few days, it eventually becomes unresponsive. When inspecting the threads of the DataNode process, I found that many of them are stuck in DataXceiver. I'd like to ask if anyone has encountered this before and if there are any recommended solutions.[root@dn-27 ~]# top -H -p 74042top - 17:24:14 up 10 days, 4:05, 1 user, load average: 140.45, 114.30, 110.42Threads: 792 total, 36 running, 756 sleeping, 0 stopped, 0 zombie%Cpu(s): 54.7 us, 38.0 sy, 0.0 ni, 7.2 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 stKiB Mem : 52732768+total, 22929336 free, 29385958+used, 21053875+buff/cacheKiB Swap: 0 total, 0 free, 0 used. 22594056+avail MemPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND56353 hdfs 20 0 11.3g 8.0g 34484 R 99.9 1.6 103:13.94 DataXceiver for72973 hdfs 20 0 11.3g 8.0g 34484 R 99.9 1.6 70:57.77 DataXceiver for84061 hdfs 20 0 11.3g 8.0g 34484 R 99.9 1.6 60:03.79 DataXceiver for11326 hdfs 20 0 11.3g

Hi All,   I have a 3 node HDP cluster and on which 2 nodes act as namenode as data node. We have recently extended the disk of HDFS from 1 tb to 3 TB. After that system got reboot and since that my name nodes are comming up.    Name node logs are mentioned below 2022-05-01 14:00:49,272 INFO namenode.FSNamesystem (FSNamesystem.java:initRetryCache(979)) - Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis2022-05-01 14:00:49,274 INFO util.GSet (LightWeightGSet.java:computeCapacity(395)) - Computing capacity for map NameNodeRetryCache2022-05-01 14:00:49,274 INFO util.GSet (LightWeightGSet.java:computeCapacity(396)) - VM type = 64-bit2022-05-01 14:00:49,275 INFO util.GSet (LightWeightGSet.java:computeCapacity(397)) - 0.029999999329447746% max memory 2.0 GB = 621.3 KB2022-05-01 14:00:49,275 INFO util.GSet (LightWeightGSet.java:computeCapacity(402)) - capacity = 2^16 = 65536 entries2022-05-01 14:00:49,299 INFO common.Storage (Storage.java

Hello all,Is it possible to update the Spark version to 3.4.x or later for trial accounts so that Spark Connect can be used? Thanks

Hello all, I have an ListS3 process that access an S3 bucket to pickup files.  This process running fine since deployed in October 2025.  Today, noticed that the files didn't get pickup and they were no error reported on the ListS3.  The state didn't list the current files in S3 waiting to pickup.  I had to stop and start the process before it get pickup.  I enabled debug to see if it will report any issue while trying to connect and also updated the run scheduled from 0 to 5 seconds. Any other suggestion or recommendation on what I can do to be prepare to help identify the issue if it occurred again in the future? Thanks!

I have installed and started Nifi.sh, it started and failed in next few seconds.nifi.properties# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.nifi.web.http.host=localhostnifi.web.http.port=8080nifi.web.http.network.interface.default=##############################################nifi.web.https.host=localhost#nifi.web.https.port=8443nifi.web.https.network.interface.default=nifi.web.https.application.protocols=h2 http/1.1nifi.web.jetty.working.directory=./work/jettynifi.web.jetty.threads=200nifi.web.max.header.size=16 KBnifi.web.proxy.context.path=nifi.web.proxy.host=nifi.web.max.content.size=nifi.web.max.requests.per.second=30000  error.log 2025-12-31 14:03:57,387 INFO [main] org.apache.nifi.bootstrap.Command Application Process [9606] started2025-12-31 14:03:57,388 INFO [main] org.apache.nifi.bootstrap.Command Bootstrap Process Running2025-12-31 14:04:57,392 INFO [StartBootstrapCommand] o.a.n.b.c.Application

I have to temporarily change permissions to several files so I'm planning to take a snapshotbefore issuing chmod commandI know that restoring a file from a snapshot is done using cp commandIn this case what does it happen, is it just restored the inode?What does it happen to unmodified files runningcp <snaproot>/.snapshot/<name>/* <target dir>/Are those file skipped?

I have a process which creates a big JSON (the content is between 5k to 400k), (and has embedded objects, arrays, arrays of objects....) my flow-file has an attribute, named ETL_ID, I want to persist the JSON to a CLOB column of an Oracle table.Well using standard PutSQL won't work as JSON content is more than 4000 chars.With some JOLT transformers I converted the content of flow file to something like:{  "ID": 1234, // ETL_ID  "REQUEST": { the very big json }}then I used PutDatabaseRecord processor to write it down, but then instead of JSON something likeMapRecord[{ContractId=null, PrintDocumentNo=1402/101/53034/53034/6, CompanyLifeProposalNo=qsn-190, InsurerId=116118216, FieldId=501, BranchId=null, AgencyId=130946, IntroducerBrokerId=null, IntroducerBranchId=null, IntroducerAgencyId=130946, BeginDate=2023-09-05, EndDate=2033-09-05, IssueDate=2023-09-05, PaymentType=233, Installment=10, InstallmentPeriod=242, Salary=null, SalaryPremiumPercentage=null, GuaranteedProfitCeiling

Hi all,I tried to start the process group using the NiFi API, but it always returned a 403 Forbidden message.I follow the steps below{{baseUrl}}/nifi-api/access/token - get the bearer token successfully{{baseUrl}}/nifi-api/process-groups/{{pgId}} passing the JSON value in the body {\"id\":\"${PG_ID}\",\"state\":\"RUNNING\"}", but I am getting 403- forbidden error message.Please help me to resolve the issue.  

Environment:NiFi Registry: [2.4.0]Deployment: Kubernetes, Helm, ZarfAuthentication: OIDC via KeycloakClient: Golang automation job running in the clusterGoal: I am running a Golang job in Kubernetes to perform initial configuration of the NiFi Registry (creating buckets/flows). The job fetches an Access Token from Keycloak  and sends it in the Authorization: Bearer header to the Registry API endpoint: GET /nifi-registry-api/access.Issue: The request fails with a 401, and the NiFi Registry logs show an IllegalArgumentException indicating an algorithm mismatch in the JWT signature validationThe logs related to this request are attached.Analysis of these logsThe Access Token issued by Keycloak is signed using RS256 (asymmetric RSA), which requires a public key to verify. However, the stack trace explicitly shows an IllegalArgumentException stating that the validation logic cannot handle asymmetric algorithms. It seems that the Registry's JwtIdentityProvider is att

We recently performed a cluster-level rolling upgrade using Cloudera Manager (CDP Private Cloud Base 7.1.9) on our High Availability (HA) enabled cluster. During the rolling restart phase, we selected:* All services (including Hive-on-Tez)* All roles (including non-worker roles)* Restart roles with stale configurations only* Restart roles with old software versions only* Redeploy client configurationDespite these selections, the Hive-on-Tez service (specifically the HiveServer2 role) was not restarted. Post-upgrade, HiveServer2 continues to show a stale configuration warning in Cloudera Manager. This behavior suggests that HiveServer2 was skipped during the rolling restart, even though it should have been included based on the selected options.We would appreciate your assistance in:1. Confirming whether this is expected behavior or a known issue.2. Advising on the correct procedure to ensure Hive-on-Tez roles are properly restarted during rolling upgrades.3. Recommending any patches or

Hello, To enable Cloudera Server High Availability, I configured a HaProxy. - Port 7183 is used for CM UI- Port 7182 is used for cloudera agentsWhile CM UI is working, there is no heartbeat from CM hosts. I got the following error from /log/cloudera-scm-agent.log [26/Nov/2025 11:51:34 +0100] 62213 MainThread agent        WARNING  Port is being used by CM Agent: 7182[26/Nov/2025 11:51:34 +0100] 62213 MainThread agent        ERROR    Heartbeating to my_load_balancer:7182 failed.Traceback (most recent call last):  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/cmf/agent.py", line 1682, in _send_heartbeat    response = self.requestor.request('heartbeat', heartbeat_data)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 137, in request    return self

Hello , I use "nifi.cmd start" to start with NIFI.Sometime  it works and most of the times it doesn't. In case  where NIFI doesn't work I am getting following logs : 2024-11-29 21:06:39,642 INFO [main] o.a.n.w.StandardContentViewerApplication Starting StandardContentViewerApplication using Java 21.0.2 with PID 24600 (D:\WorkingInstaller\2.0\nifi-2.0.0-bin\nifi-2.0.0\work\jetty\nifi-standard-content-viewer-2.0.0.war\webapp\WEB-INF\classes started by Riyaz in D:\WorkingInstaller\2.0\nifi-2.0.0-bin\nifi-2.0.0)2024-11-29 21:06:39,643 INFO [main] o.a.n.w.StandardContentViewerApplication No active profile set, falling back to 1 default profile: "default" Any pointers would be appreciated.Thanks,Riyaz