Recently active topics

HiI've got a CDE 1.22.0-h1-b3 cluster that fails to load the Spark UI after the job has completed. I can see the Spark UI during the running of a job but it fails to load/open once the job has completed.I see an http GET to the /jobs endpoint fails with a 404 Not Found.Have you come across this error and resolved it please?CheersCraig

Hello We have a pretty big flow in NiFi (NiFi 2.8.0) having many processors and my requirement is to determine how much time a particular request took. Most of the request starts with a HandleHttpRequest Processor and has some 250 processors in between them (some in the same process group and some in different process group communicating through Input and Output Ports).Sometimes we notice the request to be taking 1 minute to return the response and sometimes as less as 10 sec.So want to diagnose what is taking time.How do I determine this?I went to View Data provenance of a particular processor and then View Details and there in Details section I see some properties like-- Time, Event Duration and Lineage Duration (it is showing <1 sec for both these last two fields ). So I am not able to check accurate as it shows less than symbol. Which of these will represent the total time it took end to end (HandleHttpRequest to HandleHttpResponse). Also any rules which processor I need to do v

Hi all,I'm setting up a CML session with 64GB of RAM and 4 CPUs, then I set up a PySpark session with these configurationsspark = SparkSession.builder \.appName("OptimizedSparkSession") \.master("local[*]") \.config("spark.sql.execution.arrow.pyspark.enabled", "false") \.config("spark.sql.shuffle.partitions", "400") \.config("spark.executor.memory", "32g") \.config('spark.executor.instance', '10') \.config("spark.executor.cores", "4") \.config("spark.driver.memory", "32g") \.config("spark.network.timeout", "800s") \.config("spark.executor.heartbeatInterval", "200s") \.config("spark.driver.maxResultSize", "64g") \.getOrCreate()The transformations I do are quite simple. Basically loading two tables, one with entities of interest and another containing the additional features and other entities, so I join the second one to the first to keep only the records that are needed.table1 = spark.sql(f"""SELECT DISTINCT key1, key2FROM db1.table1""")table2 = spark.sql(f"""SELECT key1, key2, col1, c

Slide deck for my presentation at Confluent’s Current 2026. Thursday, November 5, 20262:50 PM - 3:05 PMTopic: Understanding Puffin File Format for Advanced Query Optimization in Apache Iceberg Apache Iceberg manifests store essential statistics such as min/max values and null counts, but as tables scale, query engines increasingly require richer metadata to make efficient optimization decisions. Puffin was introduced in Iceberg to address this gap by providing a file format for storing advanced statistics, sketches, and secondary indexes that do not naturally fit within manifests.In this session, we will explore the Puffin file format under the hood, including its structure, blob-based design, and snapshot-aware metadata model. We'll examine the two Puffin blob types supported today - Theta Sketches for approximate distinct counts and Deletion Vectors for row-level deletes introduced in Iceberg v3, and discuss how they enable smarter query planning, more efficient delete handling, and

We are testing Spark column-level lineage with Apache Atlas in CDP Private Cloud Base 7.1.9.A successful Spark CTAS operation creates this projection:CREATE TABLE tdc_spark_ctas_test.customers_expression_testUSING PARQUETASSELECT customer_id, first_name AS direct_first_name, upper(first_name) AS upper_first_name, concat_ws(' ', first_name, last_name) AS full_name_concat_wsFROM tdc_spark_ctas_test.customers_raw;The analyzed logical plan contains all four expressions. Atlas creates the target hive_table and all four hive_column entities, but only two spark_column_lineage entities:- customer_id -> customer_id- first_name -> direct_first_nameIt creates no spark_column_lineage entity for:- first_name -> upper_first_name- first_name + last_name -> full_name_concat_wsThe Spark application finishes successfully. Atlas receives and processes the Spark notifications with failedMessageCount: 0, and no relevant Atlas ingestion error is recorded.The CDP 7.1.9 Atlas reference

Is there a roadmap or plan whether and if yes, when the Cloudera will support dbt v2? Even a rough timeframe would be very helpful.

Hive ODBC Driver 2.10.1 only have libcurl.dll 8.21.0-DEV.how to get version 8.22 from hotfix?C:\Program Files\Cloudera ODBC Driver for Apache Hive\lib\libcurl.dll   Installed version       : 8.21.0-DEV   Fixed version           : 8.22.0   Potential Vulnerability : Component  

Hi all,I'm looking for the recommended way to avoid making the same database call for every file in NiFi.Environment- Apache NiFi version: 2.12.0- Deployment: [single node / cluster with X nodes]Use caseOur filenames include a suffix, for example: SampleFileName_SampleSuffix.xmlFor each file picked up by NiFi, we:1. Read the suffix from the filename (SampleSuffix in the example above).2. Look up that suffix in a database config table.3. Use the result to decide where the file goes, for example move it to location XYZ or location ABC.ProblemThis DB lookup runs for every file. We process a large number of files, and many of them share the same suffix, so we keep making the same DB call and getting the same result. Thanks in advance.

Dear support, community, As you all may have noticed there has been a lot of security issues and hacks being done these last few months. Exploiting vulnerabilities has never been so easy with the use of AI. Just last. week eMudhra CA certificates have been compromised. How does the apache nifi community cope with these vulnerabilities? Is there a policy for this? of a site where we can get patches or al least follow mitigation actions?  Thank you in advanced.Kind regards

Hi everyone,We’re setting up a few new worker nodes for our cluster and I’m stuck on memory. Our jobs are mostly Spark and Hive, and sometimes they run out of memory when a lot of them start at once.The servers we picked take  and I’m trying to decide how much to put in each node. Is it better to fill all the slots now, or start smaller and add more later?How much memory per worker node do you run, and has it been enough? I’d love to hear what worked for you.Thanks!

Hi all,I’d like to suggest an enhancement for Cloudera Data Engineering (CDE): a notification bell in the UI header with a badge counter that increments when jobs are executed.Current situation: When jobs run, there’s no persistent, at-a-glance indication of activity unless the job page is open. For teams that trigger jobs through pipelines or schedules, or that monitor several jobs in parallel, it’s easy to miss completions and failures.Proposed behavior:A bell icon in the CDE header with a counter of job runs with new activity since the panel was last opened. The panel lists recent runs with status (running, succeeded, failed) and duration. The counter resets when the panel is opened. Optional: alerts for failed runs or runs exceeding an expected duration.Use case: Data engineers running scheduled or pipeline-triggered jobs want to track outcomes without keeping the job page open.Has anyone seen something similar, or know if this is already on the CDE roadmap? Any pointers on the rig

Can Impala and Kudu be sold as commercial products?A Chinese  company, Sensors Data, uses them as part of their commercial product.Is this okay?

Hello,I have MsSql DB and Oracle DB. I am gonna try to explain what I want to do. I am fetching data from MsSQL DB than I am inserting that rows to Oracle DB but this structure not dynamic. Because I created TEST table on both side. So I am writing to table name of processor configuration. I want to create dynamic structure so when I start to flow, I need to migrate all database tables with all rows. But at the same time all tables need to create by processor automatically and when new rows came to MsSQL DB, the flow need to insert only new rows. Any one knows how to do that on Nifi 2.0.0-M3 ?

Hello when I try to View Data Provenance of a processor then it shows multiple rows but then when I try to view its content then I get "Content is no longer available in the content repository"though the size in KB of the input and output claims are there. Also surprisingly the new entries content I can see but the same thing does not show the next day.