Q&A

Problem statement: When i try to query iceberg table for current date which recieves data from streaming pipeline in interval of 5 mins Example: select * from <table> where result_date="<current_date>" limit 1;Error:ImpalaRuntimeException: Cannot find file in cache:: Cannot find file in cache: hdfs://xx/ya/Zzz/data/resulted/00004-22575-da5239e5-71d0-4b2f-af6b-73cbf4b7d9c5-46884-00001.parquet with snapshot id: 2154647205402518684Workaround tried: Invalidate metadata or refresh - works for few mins until next commit occurs and then throws same error with new file and new snapshot id Tried setting below as tblproperties but no helpALTER TABLE db.table_name SET TBLPROPERTIES ('metadata_refresh_interval_ms' = '60000','refresh-before-read' = 'true');Even tried to understand whether below properties have any impact but seems like no write.metadata.delete-after-commit.enabledwrite.metadata.previous-versions-maxunable to understand why this issue is poping w

Hi All,I hope you are doing well.I have already built a custom Docker image and would like to transfer and deploy it on our Cloudera AI cluster. However, the cluster does not have internet access, and I need your support on how to properly distribute and run this image in such an air-gapped environment.Could you please guide me through the best method to:Transfer the Docker image to the Cloudera cluster.Load and run it without internet access.Looking forward to your support.Best wishes,Salim

While I was trying to create a service in s3 from ranger admin I encountered an error:s3 failed to find service class org.apache.ranger.services.s3.RangerServiceS3. Resource lookup will not be available. Please make sure plugin jar is in the correct place.UsingCDH 7            7.1.9-1.cdh7.1.9.p0.44702451

I can stop all role instances...curl -v -k -u me:$pass -X POST "https://host.domain:7183/api/v54/clusters/cluster1/services/NIFI/roleCommands/stop" -H "accept: application/json" -H "Content-Type: application/json" -d "{ \"items\": [ \"NIFI_NODE1\", \"NIFI_NODE2\" ]}"...but not the role on NIFI_NODE1 only. I am trying to write a script which detects that a node has disconnected, stops the role on that host, moves the flow.xml and flow.json and then restarts the role on that host.Is there anything that lets me stop and start a role on a single host?

[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 process INFO [4545-collect-host-statistics] Instantiating process[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 process INFO [4545-collect-host-statistics] Updating process: True {}[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 process INFO First time to activate the process [4545-collect-host-statistics].[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 agent INFO Created /var/run/cloudera-scm-agent/process/4545-collect-host-statistics[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 agent INFO Chowning /var/run/cloudera-scm-agent/process/4545-collect-host-statistics to root (0) root (0)[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/4545-collect-host-statistics to 0751[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 agent INFO Created /var/run/cloudera-scm-agent/process/4545-collect-host-statistics/logs[22/Jun/2025 12:40:02 +0000] 16316 Thread-15 agent INFO Chowning /var/run/cloudera-scm-agent/process/45

https://medium.com/@surajnagendra/merge-csv-files-apache-nifi-21ba44e1b719I tried this approach but it didn't worked well

Hi everyone,I'm currently building my first Informatica mapping, which is designed to read XML documents from a Kafka topic and store them in an HDFS location.Since I'm still new to both Informatica and Cloudera, I’d appreciate your guidance on a few issues I’m facing.Setup:Cloudera version: 7.2.18 (Public Cloud)Authentication: I'm using my user keytab and a KDC/FreeIPA certificate. I’ve also created a jaas_client.conf file that allows Kafka access.This setup works fine within the Informatica Developer tool when using the files on the Informatica server.Issue 1:I'm struggling to pass these authentication files (keytab, certificate, JAAS config) to the Spark execution context so that Spark can connect to Kafka and HDFS.I manually copied the files to the /tmp directory of the master and worker nodes, but I’m unsure if this is the correct approach.Question: Is manually copying these files to Spark nodes the recommended method, or should Informatica handle this automatically when submittin

I'm setting up a single-node Cloudera Manager 7.4.4 trial on a GCP Ubuntu 20.04 VM and running into an issue during the Install Agents step. The process fails with an authentication error when trying to connect to the host.Here’s what I’ve done so far:Created a user with sudo and passwordless privilegesSet up passwordless SSH (tested with ssh $(hostname -I | awk '{print $1}') – works without password)Configured hostname and /etc/hosts (e.g., cloudera-1.localdomain)Installed OpenJDK 11 and set JAVA_HOMEStarted Cloudera Manager Server successfully (systemctl status shows it running)Port 7180 is open and accessibleUsing the .bin installer from the official CM 7.4.4 archiveDespite all this, the agent install step fails with an authentication error. Has anyone encountered this before? Is there a specific SSH or hostname nuance I might be missing?Any help would be greatly appreciated!

Hello everyone, Recently, we've decided to try out the load balancing feature in our NiFi 1.15 cluster and we've run into the following issue.16 flow files got stuck in a queue that had load balancing switched on. On the connection, I can see that there are 16 flow files and about 70 MB of data. However, when I try to list that queue, it says that it's empty and when I try to empty it, it says that 0 out of 16 flow files were removed.What's more, event after switching the load balancing feature off everywhere, it still continuously prints the following two errors in the log:Failed to communicate with Peer node001:8443 when load balancing data for Connection with ID a3f7a255-0184-1000-ffff-fffffe4ef3af due to java.io.EOFException: Encountered End-of-File when trying to verify with Peer node001:8443 whether or not space is available in Connection a3f7a255-0184-1000-ffff-fffffe4ef3afFailed to receive FlowFiles for Load Balancing due to org.apache.nifi.controller.queue.clustered.serve

Hi All, We are getting below error on windows server while establishing a connection to DB server. [Cloudera][DriverSupport] (1100) SSL certificate verification failed because the certificate is missing or incorrect. Impala ODBC driver version: 2.6.10 This was working before 2 days back and we did not make any changes to it.We have .pem file in place already.

I am trying to setup a HBase snapshot to an s3a location. This is on a closed system. Not internet accessible.I have setup these Configurations in HBaseBackup:hbase_s3_snapshot_path: /SV-INT-2020/Adavanced:hbase_service_config_safety_valveName: fs.s3a.endpointValue:https://storage-dashboard.cloud/hbase_core_site_safety_valveName: fs.s3a.endpointValue:https://storage-dashboard.cloud/When I just try to to go the Table Browser in HBase, the Recent Remote S3 Snapshots gives me this error:org.apache.hadoop.fs.s3a.AWSIOCLientException: listObjects() on s3a://SV-INT-2020/: com.amazonaws.SdkClientException: Unable to execute HTTP request: s2.amazonaws.com: Name or service not known: Unable to execute HTTP request: s2.amazonaws.com: Name or service not known:  

Hello,I have set up a 4-node NiFi 2.4.0 cluster. When I attempt to clear the queue from the NiFi UI, I encounter the following error:Node x.x.x.x:8443 is unable to fulfill this request due to: Unable to modify the data for Processor with ID d3a802c6-0196-1000-ffff-ffff90fdc7b8. Contact the system administrator.My user has admin privileges with both operate and modify permissions granted.As a workaround, I disconnected each node from the cluster one by one, cleared the queues locally on each node, and then reconnected them back to the cluster. While this temporarily resolved the issue, it is not a practical or sustainable solution.I am looking for a permanent fix to this issue. I would appreciate your assistance on this matter.

Dear All,I hope you’re doing well.I’m attempting to add a new custom runtime in Cloudera AI, but I encounter an error when clicking Validate:“Could not fetch the image metadata.”For context:I’ve already uploaded the Docker image to the Cloudera server.Running docker images | grep <image_name> confirms that the image is present.Important: The Cloudera AI server has no internet access.Could someone please help me troubleshoot this issue? Thank you in advance for your support.Best regards,Salim

I am setting up a stand-alone CDP cluster. However, getting errors when doing the agents installation steps. I can SSH to this host from another instance using both password and private key. Therefore, I can't figure out the reason.Here are the error messagesPlease let me know if I need to provide more information. Thanks in advance! 

This video (timestamped) shows you can list the queue on connections, and see provenance and lineage in flow designer: https://youtu.be/8cZJ9CyLYyI?t=5904 But in the public cloud version of Cloudera Data Flow, that functionality is missing. I can list queue and see data in many formats, but no provenance and lineage. Do we need Data Hub to do this or am I missing something? I am doing this from the Web console of Data Flow Designer.I am trying this out on a trial account of the Cloudera DataFlow, so not sure if that affects my experience. Provenance and replay was an important function I wanted to try.  Screenshots (Imgur link) 

SERVICE_MONITOR_AGGREGATION_RUN_DURATIONThe health test result for SERVICE_MONITOR_AGGREGATION_RUN_DURATION has become bad: The last metrics aggregation run duration is 33.9 second(s). Critical threshold: 30 second(s).   Service Monitor File /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR.log.out8:35:51.569 PM WARN Groups Potential performance problem: getGroups(user=hue) took 8024 milliseconds.8:36:21.270 PM WARN JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 18430ms: no GCs detected.8:36:21.558 PM WARN EnterpriseService com.cloudera.cmf.PollingScmProxy: run duration exceeded desired period. Duration: 19230 ms. Desired period: 1000 ms.8:36:22.253 PM INFO AggregatingTimeSeriesStore Run took PT33.916S which is over the slow run threshold of PT30S. 15690 metrics written for 28 entities. PT27.952S write time over 2 writes. Longest writes: PT27.876S,PT0.076S.8:37:02.720 PM INFO LDBPartition

Hi Noticed that PrometheusReportingTask is deprecated.https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-prometheus-nar/1.26.0/org.apache.nifi.reporting.prometheus.PrometheusReportingTask/index.htmlWhat alternative can we use to generate reports ? Thanks 

Hi,I am trying to fetch cloudera manager CHF patch information. I looked through the api documentation and tried to run each api methods to get the data.  I am unable to get the full patch information about Cloudera Manager in Java. Has anyone else found this information? I am looking for below information: [root@cdp719-11 ~]# rpm -qa | grep clouderacloudera-manager-daemons-7.11.3.6-50817646.el9.x86_64cloudera-manager-agent-7.11.3.6-50817646.el9.x86_64cloudera-manager-server-7.11.3.6-50817646.el9.x86_64Below is the research I did so far:Used Parcels api : gives information about parcels such as CDF - hadoop, kafka etcRan through all the api methods in the below page, none of the are giving the patch informationhttps://archive.cloudera.com/cm7/7.0.3/generic/jar/cm_api/swagger-html-sdk-docs/java/docs/ClouderaManagerResourceApi.html#collectDiagnosticDataCommand

Hi,I would like to know how can i get if it is available the company docker registry. To push the image to my company Docker registry.Best wishes,Salim 

Below is the snowflake connection url from NiFi:jdbc:snowflake://<account>.snowflakecomputing.com/?db=<db>&schema=<schema>&warehouse=<wareeehouse>user: <userid>password: <password>Driver Class Name: net.snowflake.client.jdbc.SnowflakeDriverDriver Loaction: E:\pimnifi\nifi-libsI m getting below error:Cannot create PoolableConnectionFactory (Incorrect username or password was specified.)Caused by: java.sql.SQLException: Cannot create PoolableConnectionFactory (Incorrect username or password was specified.)Caused by: net.snowflake.client.jdbc.SnowflakeSQLException: Incorrect username or password was specified.. No FlowFile to route to failure: {}Please help me in solving this error.

Hello,We have noticed that Ranger policies are no longer being enforced in HDFS and Hive in our CDP 7.3.1 cluster. Users are able to access data despite explicit deny or allow rules configured in the Ranger UI.Details:Ranger UI is up and running; all policies are visible and enabled.HDFS and Hive services do not reflect the expected access restrictions.Restarting Ranger, Hive, and HDFS services did not resolve the issue.We are not sure when the issue started.Could you please assist us in identifying why the Ranger policies are no longer being enforced in HDFS and Hive? Let us know what additional logs or configurations would help with troubleshooting.

Hi guys,i hope you are doing well.can you please help me to get the root access in the container of the session in CDWS?best wishes,salim 

I'm trying to construct a process group that produces to a Kafka topic. It's failing as all records are routed to the failure queue, but I get no errors from the PublishKafka processor to help me diagnose the problem.Digging into the PublishKafka process it seems that it doesn't produce any output whether on success or failure, despite having bulletin level set to DEBUG.If I substitute for a PutFile processor set to DEBUG then I get bulletins for each written file. So this looks like an issue specific to PublishKafka?

I'm using NiFi 1.23.2 in our production environment. It works fine when TLS is disabled, but after enabling TLS, we encounter a 'too many open files' issue. Through further investigation using lsof combined with the ss command, we discovered a large number of closed TCP connections with socket transmission buffers showing as 0t0. Why does this situation occur? Is this a bug in NiFi itself? 

Hi Team,we are using NiFi API to disable and enable all the Processors within the Processor Group. If we do it from NiFi UI it happens immediately but from API its takes 10-15 Seconds to take effect.Which is causing issue with the next API calls, currently we have used a delay of 15 sec before making next call.But is there a better way to handle this issue ?