Ask the Community

Hello,I'm trying to insert data into a hive table configured with hdfs router based federation as table locationLocation: | hdfs://router_host:8888/router/router.db/router_test_tableCluster is kebrerized and all components including Hive and RBF are working as expected except this specific use case.The hive table insert job is failing with kerberos error when using RBF as the table locationCaused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: DestHost:destPort router_host:8888 , LocalHost:localPort datanode_host. Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketForFileIdx(FileSinkOperator.java:639)at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:563)... 17 moreCaused by: java.io.IOException: DestHost:destPort router_host:8888 , LocalHost:localPort datanode_ho

Hi everyone,I'm using Hadoop 3.1.1 and have encountered an issue: after running the DataNode for a few days, it eventually becomes unresponsive. When inspecting the threads of the DataNode process, I found that many of them are stuck in DataXceiver. I'd like to ask if anyone has encountered this before and if there are any recommended solutions.[root@dn-27 ~]# top -H -p 74042top - 17:24:14 up 10 days, 4:05, 1 user, load average: 140.45, 114.30, 110.42Threads: 792 total, 36 running, 756 sleeping, 0 stopped, 0 zombie%Cpu(s): 54.7 us, 38.0 sy, 0.0 ni, 7.2 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 stKiB Mem : 52732768+total, 22929336 free, 29385958+used, 21053875+buff/cacheKiB Swap: 0 total, 0 free, 0 used. 22594056+avail MemPID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND56353 hdfs 20 0 11.3g 8.0g 34484 R 99.9 1.6 103:13.94 DataXceiver for72973 hdfs 20 0 11.3g 8.0g 34484 R 99.9 1.6 70:57.77 DataXceiver for84061 hdfs 20 0 11.3g 8.0g 34484 R 99.9 1.6 60:03.79 DataXceiver for11326 hdfs 20 0 11.3g

Hi All,   I have a 3 node HDP cluster and on which 2 nodes act as namenode as data node. We have recently extended the disk of HDFS from 1 tb to 3 TB. After that system got reboot and since that my name nodes are comming up.    Name node logs are mentioned below 2022-05-01 14:00:49,272 INFO namenode.FSNamesystem (FSNamesystem.java:initRetryCache(979)) - Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis2022-05-01 14:00:49,274 INFO util.GSet (LightWeightGSet.java:computeCapacity(395)) - Computing capacity for map NameNodeRetryCache2022-05-01 14:00:49,274 INFO util.GSet (LightWeightGSet.java:computeCapacity(396)) - VM type = 64-bit2022-05-01 14:00:49,275 INFO util.GSet (LightWeightGSet.java:computeCapacity(397)) - 0.029999999329447746% max memory 2.0 GB = 621.3 KB2022-05-01 14:00:49,275 INFO util.GSet (LightWeightGSet.java:computeCapacity(402)) - capacity = 2^16 = 65536 entries2022-05-01 14:00:49,299 INFO common.Storage (Storage.java

Hello all,Is it possible to update the Spark version to 3.4.x or later for trial accounts so that Spark Connect can be used? Thanks

Hello all, I have an ListS3 process that access an S3 bucket to pickup files.  This process running fine since deployed in October 2025.  Today, noticed that the files didn't get pickup and they were no error reported on the ListS3.  The state didn't list the current files in S3 waiting to pickup.  I had to stop and start the process before it get pickup.  I enabled debug to see if it will report any issue while trying to connect and also updated the run scheduled from 0 to 5 seconds. Any other suggestion or recommendation on what I can do to be prepare to help identify the issue if it occurred again in the future? Thanks!

I have installed and started Nifi.sh, it started and failed in next few seconds.nifi.properties# We recommend configuring HTTPS instead. The administrators guide provides instructions on how to do this.nifi.web.http.host=localhostnifi.web.http.port=8080nifi.web.http.network.interface.default=##############################################nifi.web.https.host=localhost#nifi.web.https.port=8443nifi.web.https.network.interface.default=nifi.web.https.application.protocols=h2 http/1.1nifi.web.jetty.working.directory=./work/jettynifi.web.jetty.threads=200nifi.web.max.header.size=16 KBnifi.web.proxy.context.path=nifi.web.proxy.host=nifi.web.max.content.size=nifi.web.max.requests.per.second=30000  error.log 2025-12-31 14:03:57,387 INFO [main] org.apache.nifi.bootstrap.Command Application Process [9606] started2025-12-31 14:03:57,388 INFO [main] org.apache.nifi.bootstrap.Command Bootstrap Process Running2025-12-31 14:04:57,392 INFO [StartBootstrapCommand] o.a.n.b.c.Application

I have to temporarily change permissions to several files so I'm planning to take a snapshotbefore issuing chmod commandI know that restoring a file from a snapshot is done using cp commandIn this case what does it happen, is it just restored the inode?What does it happen to unmodified files runningcp <snaproot>/.snapshot/<name>/* <target dir>/Are those file skipped?

I have a process which creates a big JSON (the content is between 5k to 400k), (and has embedded objects, arrays, arrays of objects....) my flow-file has an attribute, named ETL_ID, I want to persist the JSON to a CLOB column of an Oracle table.Well using standard PutSQL won't work as JSON content is more than 4000 chars.With some JOLT transformers I converted the content of flow file to something like:{  "ID": 1234, // ETL_ID  "REQUEST": { the very big json }}then I used PutDatabaseRecord processor to write it down, but then instead of JSON something likeMapRecord[{ContractId=null, PrintDocumentNo=1402/101/53034/53034/6, CompanyLifeProposalNo=qsn-190, InsurerId=116118216, FieldId=501, BranchId=null, AgencyId=130946, IntroducerBrokerId=null, IntroducerBranchId=null, IntroducerAgencyId=130946, BeginDate=2023-09-05, EndDate=2033-09-05, IssueDate=2023-09-05, PaymentType=233, Installment=10, InstallmentPeriod=242, Salary=null, SalaryPremiumPercentage=null, GuaranteedProfitCeiling

Hi all,I tried to start the process group using the NiFi API, but it always returned a 403 Forbidden message.I follow the steps below{{baseUrl}}/nifi-api/access/token - get the bearer token successfully{{baseUrl}}/nifi-api/process-groups/{{pgId}} passing the JSON value in the body {\"id\":\"${PG_ID}\",\"state\":\"RUNNING\"}", but I am getting 403- forbidden error message.Please help me to resolve the issue.  

Environment:NiFi Registry: [2.4.0]Deployment: Kubernetes, Helm, ZarfAuthentication: OIDC via KeycloakClient: Golang automation job running in the clusterGoal: I am running a Golang job in Kubernetes to perform initial configuration of the NiFi Registry (creating buckets/flows). The job fetches an Access Token from Keycloak  and sends it in the Authorization: Bearer header to the Registry API endpoint: GET /nifi-registry-api/access.Issue: The request fails with a 401, and the NiFi Registry logs show an IllegalArgumentException indicating an algorithm mismatch in the JWT signature validationThe logs related to this request are attached.Analysis of these logsThe Access Token issued by Keycloak is signed using RS256 (asymmetric RSA), which requires a public key to verify. However, the stack trace explicitly shows an IllegalArgumentException stating that the validation logic cannot handle asymmetric algorithms. It seems that the Registry's JwtIdentityProvider is att

We recently performed a cluster-level rolling upgrade using Cloudera Manager (CDP Private Cloud Base 7.1.9) on our High Availability (HA) enabled cluster. During the rolling restart phase, we selected:* All services (including Hive-on-Tez)* All roles (including non-worker roles)* Restart roles with stale configurations only* Restart roles with old software versions only* Redeploy client configurationDespite these selections, the Hive-on-Tez service (specifically the HiveServer2 role) was not restarted. Post-upgrade, HiveServer2 continues to show a stale configuration warning in Cloudera Manager. This behavior suggests that HiveServer2 was skipped during the rolling restart, even though it should have been included based on the selected options.We would appreciate your assistance in:1. Confirming whether this is expected behavior or a known issue.2. Advising on the correct procedure to ensure Hive-on-Tez roles are properly restarted during rolling upgrades.3. Recommending any patches or

Hello, To enable Cloudera Server High Availability, I configured a HaProxy. - Port 7183 is used for CM UI- Port 7182 is used for cloudera agentsWhile CM UI is working, there is no heartbeat from CM hosts. I got the following error from /log/cloudera-scm-agent.log [26/Nov/2025 11:51:34 +0100] 62213 MainThread agent        WARNING  Port is being used by CM Agent: 7182[26/Nov/2025 11:51:34 +0100] 62213 MainThread agent        ERROR    Heartbeating to my_load_balancer:7182 failed.Traceback (most recent call last):  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/cmf/agent.py", line 1682, in _send_heartbeat    response = self.requestor.request('heartbeat', heartbeat_data)               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^  File "/opt/cloudera/cm-agent/lib/python3.11/site-packages/avro/ipc.py", line 137, in request    return self

Hello , I use "nifi.cmd start" to start with NIFI.Sometime  it works and most of the times it doesn't. In case  where NIFI doesn't work I am getting following logs : 2024-11-29 21:06:39,642 INFO [main] o.a.n.w.StandardContentViewerApplication Starting StandardContentViewerApplication using Java 21.0.2 with PID 24600 (D:\WorkingInstaller\2.0\nifi-2.0.0-bin\nifi-2.0.0\work\jetty\nifi-standard-content-viewer-2.0.0.war\webapp\WEB-INF\classes started by Riyaz in D:\WorkingInstaller\2.0\nifi-2.0.0-bin\nifi-2.0.0)2024-11-29 21:06:39,643 INFO [main] o.a.n.w.StandardContentViewerApplication No active profile set, falling back to 1 default profile: "default" Any pointers would be appreciated.Thanks,Riyaz

Hello everyone,I have installed Apache NiFi 2.7.1 on RHEL9. Unfortunately, I can't access it via a web browser (port 8443) from another computer.The firewall in RHEL is configured for port 8443, and even disabling the firewall doesn't allow access.The following entries appear in the nifi-app-log:2025-12-17 13:27:51,876 INFO [main] org.eclipse.jetty.server.Server Started oejs.Server@6f25bf88{STARTING}[12.1.5,sto=0] @11898ms2025-12-17 13:27:51,877 INFO [main] org.apache.nifi.web.server.JettyServer Started Server on https://vm-P004-4067:8443/nifi 2025-12-17 13:27:51,883 INFO [main] o.a.n.runtime.StandardManagementServer Started Management Server on http://127.0.0.1:52020 2025-12-17 13:27:51,884 INFO [main] org.apache.nifi.runtime.Application Started Application in 15.213 seconds (15213111810 ns)In the nifi.properties file, I only specified the hostname of the Linux server in the following entry: nifi.web.https.host.Does anything else need to be configured? All the example videos

My company does not allow Java any more. Does the Impala JDBC driver work with alternatives, such as Zing?Thanks,Kevin

Hi All, I am able to launch the Oozie work flow using Oozie Webservice API  via CURL utility .Like about perform HTTP GET and POST requests . Example curl command:curl -u guestuser:guestpwd  -X POST -H "Content-Type: application/xml;charset=UTF-8" -d @Former Member.xml "https://knowxhost:8443/gateway/cdp-proxy-api/oozie/v1/jobs?action=start" url: https://knowxhost:8443/gateway/cdp-proxy-api/oozie/v1/job/0000009-210910123241282-oozie-oozi-W'Also in python , I can fetch the Oozie executed workflow.    response = requests.get(url, auth=HTTPBasicAuth('guestuser','guestpwd '), verify=False).text     But unable to perform the POST request and getting 500 Error . url = 'https://knowxhost:8443/gateway/cdp-proxy-api/oozie/v1/jobs?action=start'    response = requests.post(url, auth=HTTPBasicAuth('guestuser','guestpwd'), verify=False, data=xml, headers=headers).text    Exception

Please find the below screenshot and attached logs. Error:SEVERE: The Integration Service failed to run the Hive task [Write_test_1jiq2sdd]. See the additional error messages for more information.com.informatica.sdk.dtm.ExecutionException: [[HIVE_1070] The Integration Service failed to run Hive query [Write_test_1jiq2sdd_query_3] for task [Write_test_1jiq2sdd] due to following error: Hive error code [2], Hive message [Error while compiling statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1764063775281_0054_1_00, diagnostics=[Vertex vertex_1764063775281_0054_1_00 [Map 1] killed/failed due to:INIT_FAILURE, Fail to create InputInitializerManager, java.util.concurrent.ExecutionException: org.apache.tez.dag.api.TezReflectionException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator  2025-11-26 14:25:24,278 AST <LdtmWorkflowT

Hello!NiFi version - 2.5.0Java - 21When trying to run ExecuteSQL processor i'm getting error:What does it mean and where is this unsupported field located? Log:It's something wrong with java or nifi or whole system?Please, help me to figure it out 

Intel's 2003 patent: Method and apparatus for parallel table lookup using SIMD instructionshttps://patents.google.com/patent/US20040054879A1/enIn July 2006, Intel noted that PSHUFB was available on the Core 2 Duo CPUs, such as the X6800, to some companies and universities working with error correction code. I recall some published programs that were using it later on that year (around September 2006). However I've seen one or more later patents (2010 or later) that include using PSHUFB for parallel table lookup, which doesn't make sense. 

I have multiple instances of nifi running and a central nifi registry is deployed. NiFi Registry is signed by an independent self-issued CA-signed certificate.I do have a copy of NiFi Registry's truststore and keystore inside my NiFi instance (Kubernetes Pod). And I have configured the registry client with my registry's internal k8s service URL and the associated SSL Context is referenced to the Registry's truststore and keystores. The SSL Context Controller enables fine (indicating the keystores are accessible and the passwords are correct) but then I get the following error message while trying to enable versioning on a process group:Unable to obtain listing of buckets: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested targetI checked the other topics on this forum and on other forums on the Internet and found that this could be an issue with the validity of the registry's server certificate. Th

Hi,So I have a standard stand alone install of nifi 2.5.0 on windows 10 ( no LDAP etc ) which created a generated username and password, and which allows me to log in fine.What I want to do is change this existing nifi generated user from being a normal user to an administrator. Its so i can create new processors via the api.  Apparently on the very first login to a new nifi instance it logs you in as admin, but I tried this and couldnt see any options on the menu etc that looked like I had admin privs. But when it starts and runs, it uses port 8400, so its possible someone in the company may have modified the set up slightly? When it starts for the first time it goes through the expansion process.Can someone please suggest the steps to change the generated normal user to admin level please, or if maybe I'm going about this the wrong way?  I have consulted the following doco but no joy. https://nifi.apache.org/nifi-docs/administration-guide.html#creating-users-groupsI have sp

 Error importing NiFi workflow template from version 1.x to 2.xI'm having trouble migrating my NiFi workflow from version 1.9.2 to version 2.0 using the export template and import template functions. Any assistance would be greatly appreciated.

Hello,I've installed nifi 2.6 registry security - then I've scanned it in AWS Inspector, it shows me the following results:0 Critical.12 High.12 Medium.Could anyoune confrim the results? And if this is a stable security version?

Hello there, I'm having a problem while trying to start a newly added node's yarn NodeMgr.When I start it via Cloudera Manager console, I've got this in logs:2025-12-09 10:15:52,431 ERROR org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Unexpected error starting NodeStatusUpdaterorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: Received SHUTDOWN signal from Resourcemanager, Registration of NodeManager failed, Message from ResourceManager: Disallowed NodeManager from cloudera06apl.claro.amx, Sending SHUTDOWN signal to the NodeManager.at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.registerWithRM(NodeStatusUpdaterImpl.java:426)at org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl.serviceStart(NodeStatusUpdaterImpl.java:272)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:121)at org.apache.hadoop.service.AbstractService.start(Abs

Hi Team,We are on CDP 7.1.9 SP1 and on OS version RHEL 7. As RHEL 7 is EOL we are planning to upgrade to RHEL 8 and also mysql 5.7 to 8.  Since it is production environment we are little bit concerned how complex it would be. Our CM services and mysql database of cm and application are residing on the same node.Kindly suggest apart from taking mysqldump backups and exporting the database/table what else we need to take care of post OS upgrade to rhel 8, I mean I am aware of re-installing el8 cm packages but for mysql we have to install from rpms of el8 so finding it little tricky.