Q&A

Hi all,I'm looking for the recommended way to avoid making the same database call for every file in NiFi.Environment- Apache NiFi version: 2.12.0- Deployment: [single node / cluster with X nodes]Use caseOur filenames include a suffix, for example: SampleFileName_SampleSuffix.xmlFor each file picked up by NiFi, we:1. Read the suffix from the filename (SampleSuffix in the example above).2. Look up that suffix in a database config table.3. Use the result to decide where the file goes, for example move it to location XYZ or location ABC.ProblemThis DB lookup runs for every file. We process a large number of files, and many of them share the same suffix, so we keep making the same DB call and getting the same result. Thanks in advance.

We are testing Spark column-level lineage with Apache Atlas in CDP Private Cloud Base 7.1.9.A successful Spark CTAS operation creates this projection:CREATE TABLE tdc_spark_ctas_test.customers_expression_testUSING PARQUETASSELECT customer_id, first_name AS direct_first_name, upper(first_name) AS upper_first_name, concat_ws(' ', first_name, last_name) AS full_name_concat_wsFROM tdc_spark_ctas_test.customers_raw;The analyzed logical plan contains all four expressions. Atlas creates the target hive_table and all four hive_column entities, but only two spark_column_lineage entities:- customer_id -> customer_id- first_name -> direct_first_nameIt creates no spark_column_lineage entity for:- first_name -> upper_first_name- first_name + last_name -> full_name_concat_wsThe Spark application finishes successfully. Atlas receives and processes the Spark notifications with failedMessageCount: 0, and no relevant Atlas ingestion error is recorded.The CDP 7.1.9 Atlas reference

Hive ODBC Driver 2.10.1 only have libcurl.dll 8.21.0-DEV.how to get version 8.22 from hotfix?C:\Program Files\Cloudera ODBC Driver for Apache Hive\lib\libcurl.dll   Installed version       : 8.21.0-DEV   Fixed version           : 8.22.0   Potential Vulnerability : Component  

Dear support, community, As you all may have noticed there has been a lot of security issues and hacks being done these last few months. Exploiting vulnerabilities has never been so easy with the use of AI. Just last. week eMudhra CA certificates have been compromised. How does the apache nifi community cope with these vulnerabilities? Is there a policy for this? of a site where we can get patches or al least follow mitigation actions?  Thank you in advanced.Kind regards

Hi everyone,We’re setting up a few new worker nodes for our cluster and I’m stuck on memory. Our jobs are mostly Spark and Hive, and sometimes they run out of memory when a lot of them start at once.The servers we picked take  and I’m trying to decide how much to put in each node. Is it better to fill all the slots now, or start smaller and add more later?How much memory per worker node do you run, and has it been enough? I’d love to hear what worked for you.Thanks!

Hi all,I’d like to suggest an enhancement for Cloudera Data Engineering (CDE): a notification bell in the UI header with a badge counter that increments when jobs are executed.Current situation: When jobs run, there’s no persistent, at-a-glance indication of activity unless the job page is open. For teams that trigger jobs through pipelines or schedules, or that monitor several jobs in parallel, it’s easy to miss completions and failures.Proposed behavior:A bell icon in the CDE header with a counter of job runs with new activity since the panel was last opened. The panel lists recent runs with status (running, succeeded, failed) and duration. The counter resets when the panel is opened. Optional: alerts for failed runs or runs exceeding an expected duration.Use case: Data engineers running scheduled or pipeline-triggered jobs want to track outcomes without keeping the job page open.Has anyone seen something similar, or know if this is already on the CDE roadmap? Any pointers on the rig

Can Impala and Kudu be sold as commercial products?A Chinese  company, Sensors Data, uses them as part of their commercial product.Is this okay?

Hello,I have MsSql DB and Oracle DB. I am gonna try to explain what I want to do. I am fetching data from MsSQL DB than I am inserting that rows to Oracle DB but this structure not dynamic. Because I created TEST table on both side. So I am writing to table name of processor configuration. I want to create dynamic structure so when I start to flow, I need to migrate all database tables with all rows. But at the same time all tables need to create by processor automatically and when new rows came to MsSQL DB, the flow need to insert only new rows. Any one knows how to do that on Nifi 2.0.0-M3 ?

Hello when I try to View Data Provenance of a processor then it shows multiple rows but then when I try to view its content then I get "Content is no longer available in the content repository"though the size in KB of the input and output claims are there. Also surprisingly the new entries content I can see but the same thing does not show the next day.

I am getting an error when using the ExcelReader to convert an Excel sheet to JSON. Everything works except for some dates. If I remove the Date Format setting on the controller, no issue, but get the date in milliseconds from 1/1/1970, which I do not want. The ExcelReader is failing on the date 05/01/2017. This is a valid MM/dd/yyyy date. The error I get is: ERROR015de330-01a0-1000-0000-00001cb4e04eAll NodesConvertRecord[id=015de330-01a0-1000-0000-00001cb4e04e] Failed to process FlowFile[filename=Test_File.xlsx]; will route to failure: org.apache.nifi.processor.exception.ProcessException: Could not parse incoming data- Caused by: org.apache.nifi.serialization.MalformedRecordException: Read next Record from Excel XLSX failed on row 59 in sheet Data- Caused by: org.apache.nifi.serialization.record.field.FieldConversionException: Conversion failed for [05/01/2017] named [column_9] to [java.time.LocalDate] [java.time.format.DateTimeParseException] Text '05/01/2017' could not be parse

Hello,I have a 3 nodes cluster environment running Nifi version 1.29.  Nifi logs are sync to Elastic Search; I noticed that GetSFTP is reporting Processing Failed multiple times.  It appears the processor recover on its own for example from January 1st to present reported 392 times events.   I'm trying to identify, why this is occurring as the log doesn't show much beside that message.    Any idea why this is occurring?Example of message:GetSFTP[id=633332e2-0ad4-17db-85d0-2ec76b7a49bc] Processing failedThanks!

HelloHow to read the body of the mail in nifi? I don't see any relevant processor for this.I see there is an "ExtractEmailAttachments" processor but will it read body reliably? We tried to test and see that it is able to make two flowfiles after executing. One is with a text/plain and has the email body present and another one is text/html having some html content having what we sent in mail as well.But due to its name being "ExtractEmailAttachment" and not something like ExtractEmailBody I am not sure.How to effectively and reliably read the mail body? In NiFi 1.28 we had Python (which was able to read the body of mail) but in NiFi 2.8 we don't have Python and able to see GroovyScript which is not allowing import of javax.mail type libraries which is required for writing the logic to fetch the email body.

Hi There, We are currently using Cloudera On-Prem (CDP 7.1.9SP1, CFM 2.1.7, CM 7.13.1), where monitoring of Spark jobs (such as memory utilization, CPU usage, active jobs, completed jobs, and related metrics) appears to be primarily available through the UI.We explored several approaches for retrieving Spark Structured Streaming and job execution metrics programmatically. However, most publicly available documentation references accessing the Spark Driver UI via port 4040 (same case with 4041,4042,4043 etc) on the driver node. In our Cloudera environment, this endpoint does not appear to be operational or accessible, which limits our ability to leverage those documented methods.Could you please point us in the right direction or suggest alternative approaches for retrieving such monitoring data through command-line interfaces, APIs, logs, event stores, or any other available mechanisms within Cloudera?At this stage, we are not looking for a specific metric alone. Our objective is

Hi all — we're migrating from NiFi Registry to the git-based Flow Registry Client, specifically the BitbucketFlowRegistryClient (NiFi 2.11.0), and validating the workflow.From what we've seen so far, committing a flow version via the NiFi UI pushes directly to the configured Bitbucket branch, with no built-in approval step — any review has to happen externally via a Bitbucket PR after the fact.Questions for the community:Approval workflow: Is there any NiFi-native mechanism (now or planned) to gate a version commit behind a review/approval, or is external git-host PR review the intended pattern?Branching: For teams enforcing PR-based review on flow changes, do you commit to a feature branch by default and require manual PR creation, or is there tooling to automate opening the PR after a NiFi commit?Diffing: Any recommended pattern for diffing flow JSON in PRs on Bitbucket specifically? We saw Datavolo's Flow Diff GitHub Action for GitHub — is there an equivalent for Bitbucket, or does

Hello.Before deleting rows in a specific table (463,462 rows in table), HDFS file size is :$ hadoop fs -du -s -h /apps/hive/warehouse/prd_thmil.db/th_mil_fb_code_value_brut54.2 M 162.6 M /apps/hive/warehouse/prd_thmil.db/th_mil_fb_code_value_brut54.2 Mb is the size of 1 file and each file is replicated 2 times so 162.6 Mb is the total size, it OK.But after deleting more than 450,000 rows in the table (12,890 rows remaining after the DELETE), the file size didn't change at all.Is it normal ? When new rows are added in the table, file size won't grow and HDFS will 'overwrite' older data with the new one ?Regards

I'm running several ETL flows in CDF/Nifi. The flows all involve writing some data to Iceberg tables. These flows are mostly running on a schedule that ranges from hourly to weekly (i.e., infrequent writes). Each data flow deployment has a few independent DAGs that all end with the PutIceberg processor, such that there are multiple separate ETL processes running at different intervals all in the same Nifi deployment.The problem I see occurs pretty rarely (~once a month) and only for some deployments. Occasionally, the PutIceberg processor will error with 'Failed to acquire a SAS token' (see full error log below).This will continue happening every time the processor runs unless I restart the flow and then it will run fine again. It seems to happen more often when the processor runs at a daily interval. My flows are using Nifi runtime 1.27.0.2.3.14.0-14 and I'm on CDP Public Cloud on Azure.```org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException: Failed to acq

Apache NiFi 2.6.0 upstream ships with Angular 19.2.14. Apache did not change the Angular dependency within the NiFi 2.6.x release line. Instead, the Angular dependency update was introduced as part of the NiFi 2.7.0 development cycle.In contrast, CFM modified the Angular dependency while continuing to ship NiFi 2.6.0.CFM 4.12.0.0 upgraded Angular from 19.2.14 to 19.2.19. After this change, the frontend no longer sent the Request-Token header with API requests, causing NiFi’s XSRF protection to reject legitimate UI operations. This regression was later addressed in CFM 4.12.0.1.However, instead of remaining on the Angular 19.x patch line, CFM 4.12.0.1 upgraded Angular again to 20.2.4.After inspecting the compiled frontend bundled with CFM 4.12.0.1, I confirmed that it is built with Angular 20.2.4. I also verified that the bundled frontend still follows the pre-fix implementation for GHSA-58c5-g7wp-6w37 and does not contain the upstream security fix. According to Angular’s official secur

I have been trying to send data from Kafka to snowflake using the JDBC driver with Kafka connect. some details about the environment:Kafka is running in a Cloudera private cluster (Base 7.1.9).The version of the Snowflake-kafka connector driver is 2.2.2.The configuration of the Kafka (v3.2.0+) connector is:{ "buffer.count.records": "2", "buffer.flush.time": "60", "buffer.size.bytes": "5000000", "connector.class": "com.snowflake.kafka.connector.SnowflakeSinkConnector", "errors.log.enable": "true", "errors.log.include.messages": "true", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "snowflake.database.name": "kafka_db", "snowflake.private.key": "REDACTED", "snowflake.schema.name": "kafka_schema", "snowflake.topic2table.map": "snowflake_kafka_poc:existing_table1", "snowflake.url.name": "https://REDACTED.snowflakecomputing.com", "snowflake.user.name": "kafka_connector_user_1", "tasks.max": "4", "topics": "snowflake_kafka_poc", "value.converter": "com.

Hi all,We've upgraded our base cluster (Cloudera Runtime 7.1.9) to CDS 3.5via the new add-on parcel — that part is working fine.Now we're trying to make Spark 3.5 available as a selectable RuntimeAddon inside Cloudera AI Workbench sessions, but we're stuck.Our setup:- Cloudera AI Workbench version: 2.0.44-h1-b4- CDP Private Cloud Data Services: 1.5.4-h10-b2- Deployment type: ECS- We use Spark Pushdown (sessions submit jobs that run on the basecluster via YARN)What we've found so far:- Currently only Spark versions up to 3.3.2 show up in SiteAdministration > Runtime > Runtime Addons- There's a SPARK3 3.5.4 parcel visible in Cloudera Manager underour CAI cluster, listed as "Available Remotely" — not yetdownloaded/activatedHas anyone gotten Spark 3.5 working as a Runtime Addon on a similarsetup? Specifically:1. Is downloading/activating that SPARK3 parcel the right approach,or does it need to be "linked" to the workbench some other way?2. Does this require a minimum Workbench versio

PVC 서비스 재시작 후 'Pod Count' 서비스 재시작이전보다 감소하는 증상이 발생하여 정상유무를 확인하고자문의를 드립니다.+ 첨부 그림 파일처럼 서비스 재시작전에는 "Pod Cont" 갯수가 54개로 표시되지만, 서비스 재시작 후 18개로 줄어들어서 해당 증상이 비정상일 경우 확인해야 되는 항목과 참고할 수 있는 로그정보가 필요합니다.'Pod Count'에는 어떤 정보가 포함되어 있나요? 

Hi, Recently in one of our critical Cloudera CDP 7.1.7_p1044 cluster we could see that Impala is getting hanged intermittently and whenever Impala session are getting hanged no one was being able to run even Show databases and all Impala sessions are getting hanged. While restarting the Impala Catalog service the issue gets resolved but again it reappears after 5-6 hours or may be after 1 days as there is no fixed time.We checked the Impala catalog log and found the following errors when it is in hung state: "Error getting policies; Received NULL response!!. secure Mode=true, user=impala/pc1133.aaa.abc.com@corp.xxx.COM (auth: KERBEROS), service Name=hive" Also while checking the Ranger errors we got the following error : "RangerAdminRESTClient.java:167] Error getting policies; Received NULL response!!" Also getting error in Hive Server 2 logs: "Error occurred during processing of message.SSL Handshake Exception: Remote host terminated the handshake" A

I have copied hive table from Prod to QA by using get and put command but while executing select counnt (*) is it showing 0 count but i can see physical location is having data however i have checked all permission are in place so i did invalidate metadata and table refresh and msck repiare but still facing same issue. can seomeone help. RegardsSaim

Hello TeamWe have an increasing number of applications being added to our NiFi server. Now the nifi has got too slow that it won't load the home page or when we click a process group then it won't open despite browser url showing the process group id as if page should have gone there.Also the apis we have implemented using HandleHttpRequest return Socket Timeout error.Please note these were working fine earlier. What should we do?

I have set my impala.admission-control.min-query-mem-limit.root.default to 2GiB and my impala.admission-control.max-query-mem-limit.root.default to 57GiB, which aligned with the memory allocation for each Impala Executor node that I've set on the resource templates. My problem is, when I'm running a heavy query (I tried to set it so the limit is 10000000), the limit memory is set to 2GiB and makes the query OOM, which is strange because the max-query-mem-limit is set to 57GiB, so the limit should be > 2GiB and <= 57GiB if the query is heavy, right? Yet it's just set to the minimum mem limit. I know that the limit is set to clamp(min memory limit per query, memory estimated, max memory limit per query), but the estimated memory is 4GB and what makes it stranger is that when I try to define SET MEM_LIMIT in the query to another number like 4G or 8G, the estimated memory changes; which means there's some calculation for the estimated memory based on the memory li

I missed my CDP-0011 certification appointment. What should I do?The certification fee is very expensive for me, so I cannot repurchase it. Has this situation happened to any of you, and what is the correct action I should take?