Q&A

Hello community,I have a group of NiFi clusters managed by the same Cloudera Manager. I have been having RegionTooBusyExceptions related to PutHBaseRecord processors in one of those clusters.I see that the parallelPutToStoreThreadLimit property is set to 10 by default. I would like to increase this amount only for the cluster that is having the issue, not for the entire Cloudera environment.I have already tried updating the hbase.region.store.parallel.put.limit property in the hbase-site.xml files stored in the cluster that is having the issue, but that did not work and the error persists.Is there any way to configure or override this limit exclusively from the client side for a specific cluster? Or is this strictly a global server-side setting that cannot be changed per client?Thanks in advance!

Is there a roadmap or plan whether and if yes, when the Cloudera will support dbt v2? Even a rough timeframe would be very helpful.

Hi,I am using Apache NiFi 2.9 running in Kubernetes and need to connect a ConsumeKafka processor to Google Cloud Managed Service for Apache Kafka using Kafka3ConnectionService.Due to a security constraint, SASL/PLAIN cannot be used. I need to use:Security Protocol: SASL_SSLSASL Mechanism: OAUTHBEARERFor Google Cloud Managed Kafka authentication, I am trying to use Google's Kafka authentication callback handler with Application Default Credentials (ADC).The authentication setup requires the Google/Kafka authentication dependencies, including: google-auth-library-oauth2-http-1.47.0.jarkafka-oauth-client-0.17.1.jarmanaged-kafka-auth-login-handler-1.0.5.jarI have made these dependencies available to the NiFi environment.I have also configured NiFi to use a JAAS configuration file through bootstrap.conf:.properties  java.arg.18=-Djava.security.auth.login.config=/opt/nifi/nifi-current/conf/kafka_client_jaas.confThe JAAS configuration uses: org.apache.kafka.common.security.oauthbearer.OAuthBe

HiI've got a CDE 1.22.0-h1-b3 cluster that fails to load the Spark UI after the job has completed. I can see the Spark UI during the running of a job but it fails to load/open once the job has completed.I see an http GET to the /jobs endpoint fails with a 404 Not Found.Have you come across this error and resolved it please?CheersCraig

Hello We have a pretty big flow in NiFi (NiFi 2.8.0) having many processors and my requirement is to determine how much time a particular request took. Most of the request starts with a HandleHttpRequest Processor and has some 250 processors in between them (some in the same process group and some in different process group communicating through Input and Output Ports).Sometimes we notice the request to be taking 1 minute to return the response and sometimes as less as 10 sec.So want to diagnose what is taking time.How do I determine this?I went to View Data provenance of a particular processor and then View Details and there in Details section I see some properties like-- Time, Event Duration and Lineage Duration (it is showing <1 sec for both these last two fields ). So I am not able to check accurate as it shows less than symbol. Which of these will represent the total time it took end to end (HandleHttpRequest to HandleHttpResponse). Also any rules which processor I need to do v

Hi all,I'm setting up a CML session with 64GB of RAM and 4 CPUs, then I set up a PySpark session with these configurationsspark = SparkSession.builder \.appName("OptimizedSparkSession") \.master("local[*]") \.config("spark.sql.execution.arrow.pyspark.enabled", "false") \.config("spark.sql.shuffle.partitions", "400") \.config("spark.executor.memory", "32g") \.config('spark.executor.instance', '10') \.config("spark.executor.cores", "4") \.config("spark.driver.memory", "32g") \.config("spark.network.timeout", "800s") \.config("spark.executor.heartbeatInterval", "200s") \.config("spark.driver.maxResultSize", "64g") \.getOrCreate()The transformations I do are quite simple. Basically loading two tables, one with entities of interest and another containing the additional features and other entities, so I join the second one to the first to keep only the records that are needed.table1 = spark.sql(f"""SELECT DISTINCT key1, key2FROM db1.table1""")table2 = spark.sql(f"""SELECT key1, key2, col1, c

We are testing Spark column-level lineage with Apache Atlas in CDP Private Cloud Base 7.1.9.A successful Spark CTAS operation creates this projection:CREATE TABLE tdc_spark_ctas_test.customers_expression_testUSING PARQUETASSELECT customer_id, first_name AS direct_first_name, upper(first_name) AS upper_first_name, concat_ws(' ', first_name, last_name) AS full_name_concat_wsFROM tdc_spark_ctas_test.customers_raw;The analyzed logical plan contains all four expressions. Atlas creates the target hive_table and all four hive_column entities, but only two spark_column_lineage entities:- customer_id -> customer_id- first_name -> direct_first_nameIt creates no spark_column_lineage entity for:- first_name -> upper_first_name- first_name + last_name -> full_name_concat_wsThe Spark application finishes successfully. Atlas receives and processes the Spark notifications with failedMessageCount: 0, and no relevant Atlas ingestion error is recorded.The CDP 7.1.9 Atlas reference

Hive ODBC Driver 2.10.1 only have libcurl.dll 8.21.0-DEV.how to get version 8.22 from hotfix?C:\Program Files\Cloudera ODBC Driver for Apache Hive\lib\libcurl.dll   Installed version       : 8.21.0-DEV   Fixed version           : 8.22.0   Potential Vulnerability : Component  

Hi all,I'm looking for the recommended way to avoid making the same database call for every file in NiFi.Environment- Apache NiFi version: 2.12.0- Deployment: [single node / cluster with X nodes]Use caseOur filenames include a suffix, for example: SampleFileName_SampleSuffix.xmlFor each file picked up by NiFi, we:1. Read the suffix from the filename (SampleSuffix in the example above).2. Look up that suffix in a database config table.3. Use the result to decide where the file goes, for example move it to location XYZ or location ABC.ProblemThis DB lookup runs for every file. We process a large number of files, and many of them share the same suffix, so we keep making the same DB call and getting the same result. Thanks in advance.

Dear support, community, As you all may have noticed there has been a lot of security issues and hacks being done these last few months. Exploiting vulnerabilities has never been so easy with the use of AI. Just last. week eMudhra CA certificates have been compromised. How does the apache nifi community cope with these vulnerabilities? Is there a policy for this? of a site where we can get patches or al least follow mitigation actions?  Thank you in advanced.Kind regards

Hi everyone,We’re setting up a few new worker nodes for our cluster and I’m stuck on memory. Our jobs are mostly Spark and Hive, and sometimes they run out of memory when a lot of them start at once.The servers we picked take  and I’m trying to decide how much to put in each node. Is it better to fill all the slots now, or start smaller and add more later?How much memory per worker node do you run, and has it been enough? I’d love to hear what worked for you.Thanks!

Hi all,I’d like to suggest an enhancement for Cloudera Data Engineering (CDE): a notification bell in the UI header with a badge counter that increments when jobs are executed.Current situation: When jobs run, there’s no persistent, at-a-glance indication of activity unless the job page is open. For teams that trigger jobs through pipelines or schedules, or that monitor several jobs in parallel, it’s easy to miss completions and failures.Proposed behavior:A bell icon in the CDE header with a counter of job runs with new activity since the panel was last opened. The panel lists recent runs with status (running, succeeded, failed) and duration. The counter resets when the panel is opened. Optional: alerts for failed runs or runs exceeding an expected duration.Use case: Data engineers running scheduled or pipeline-triggered jobs want to track outcomes without keeping the job page open.Has anyone seen something similar, or know if this is already on the CDE roadmap? Any pointers on the rig

Can Impala and Kudu be sold as commercial products?A Chinese  company, Sensors Data, uses them as part of their commercial product.Is this okay?

Hello,I have MsSql DB and Oracle DB. I am gonna try to explain what I want to do. I am fetching data from MsSQL DB than I am inserting that rows to Oracle DB but this structure not dynamic. Because I created TEST table on both side. So I am writing to table name of processor configuration. I want to create dynamic structure so when I start to flow, I need to migrate all database tables with all rows. But at the same time all tables need to create by processor automatically and when new rows came to MsSQL DB, the flow need to insert only new rows. Any one knows how to do that on Nifi 2.0.0-M3 ?

Hello when I try to View Data Provenance of a processor then it shows multiple rows but then when I try to view its content then I get "Content is no longer available in the content repository"though the size in KB of the input and output claims are there. Also surprisingly the new entries content I can see but the same thing does not show the next day.

I am getting an error when using the ExcelReader to convert an Excel sheet to JSON. Everything works except for some dates. If I remove the Date Format setting on the controller, no issue, but get the date in milliseconds from 1/1/1970, which I do not want. The ExcelReader is failing on the date 05/01/2017. This is a valid MM/dd/yyyy date. The error I get is: ERROR015de330-01a0-1000-0000-00001cb4e04eAll NodesConvertRecord[id=015de330-01a0-1000-0000-00001cb4e04e] Failed to process FlowFile[filename=Test_File.xlsx]; will route to failure: org.apache.nifi.processor.exception.ProcessException: Could not parse incoming data- Caused by: org.apache.nifi.serialization.MalformedRecordException: Read next Record from Excel XLSX failed on row 59 in sheet Data- Caused by: org.apache.nifi.serialization.record.field.FieldConversionException: Conversion failed for [05/01/2017] named [column_9] to [java.time.LocalDate] [java.time.format.DateTimeParseException] Text '05/01/2017' could not be parse

Hello,I have a 3 nodes cluster environment running Nifi version 1.29.  Nifi logs are sync to Elastic Search; I noticed that GetSFTP is reporting Processing Failed multiple times.  It appears the processor recover on its own for example from January 1st to present reported 392 times events.   I'm trying to identify, why this is occurring as the log doesn't show much beside that message.    Any idea why this is occurring?Example of message:GetSFTP[id=633332e2-0ad4-17db-85d0-2ec76b7a49bc] Processing failedThanks!

HelloHow to read the body of the mail in nifi? I don't see any relevant processor for this.I see there is an "ExtractEmailAttachments" processor but will it read body reliably? We tried to test and see that it is able to make two flowfiles after executing. One is with a text/plain and has the email body present and another one is text/html having some html content having what we sent in mail as well.But due to its name being "ExtractEmailAttachment" and not something like ExtractEmailBody I am not sure.How to effectively and reliably read the mail body? In NiFi 1.28 we had Python (which was able to read the body of mail) but in NiFi 2.8 we don't have Python and able to see GroovyScript which is not allowing import of javax.mail type libraries which is required for writing the logic to fetch the email body.

Hi There, We are currently using Cloudera On-Prem (CDP 7.1.9SP1, CFM 2.1.7, CM 7.13.1), where monitoring of Spark jobs (such as memory utilization, CPU usage, active jobs, completed jobs, and related metrics) appears to be primarily available through the UI.We explored several approaches for retrieving Spark Structured Streaming and job execution metrics programmatically. However, most publicly available documentation references accessing the Spark Driver UI via port 4040 (same case with 4041,4042,4043 etc) on the driver node. In our Cloudera environment, this endpoint does not appear to be operational or accessible, which limits our ability to leverage those documented methods.Could you please point us in the right direction or suggest alternative approaches for retrieving such monitoring data through command-line interfaces, APIs, logs, event stores, or any other available mechanisms within Cloudera?At this stage, we are not looking for a specific metric alone. Our objective is

Hi all — we're migrating from NiFi Registry to the git-based Flow Registry Client, specifically the BitbucketFlowRegistryClient (NiFi 2.11.0), and validating the workflow.From what we've seen so far, committing a flow version via the NiFi UI pushes directly to the configured Bitbucket branch, with no built-in approval step — any review has to happen externally via a Bitbucket PR after the fact.Questions for the community:Approval workflow: Is there any NiFi-native mechanism (now or planned) to gate a version commit behind a review/approval, or is external git-host PR review the intended pattern?Branching: For teams enforcing PR-based review on flow changes, do you commit to a feature branch by default and require manual PR creation, or is there tooling to automate opening the PR after a NiFi commit?Diffing: Any recommended pattern for diffing flow JSON in PRs on Bitbucket specifically? We saw Datavolo's Flow Diff GitHub Action for GitHub — is there an equivalent for Bitbucket, or does

Hello.Before deleting rows in a specific table (463,462 rows in table), HDFS file size is :$ hadoop fs -du -s -h /apps/hive/warehouse/prd_thmil.db/th_mil_fb_code_value_brut54.2 M 162.6 M /apps/hive/warehouse/prd_thmil.db/th_mil_fb_code_value_brut54.2 Mb is the size of 1 file and each file is replicated 2 times so 162.6 Mb is the total size, it OK.But after deleting more than 450,000 rows in the table (12,890 rows remaining after the DELETE), the file size didn't change at all.Is it normal ? When new rows are added in the table, file size won't grow and HDFS will 'overwrite' older data with the new one ?Regards

I'm running several ETL flows in CDF/Nifi. The flows all involve writing some data to Iceberg tables. These flows are mostly running on a schedule that ranges from hourly to weekly (i.e., infrequent writes). Each data flow deployment has a few independent DAGs that all end with the PutIceberg processor, such that there are multiple separate ETL processes running at different intervals all in the same Nifi deployment.The problem I see occurs pretty rarely (~once a month) and only for some deployments. Occasionally, the PutIceberg processor will error with 'Failed to acquire a SAS token' (see full error log below).This will continue happening every time the processor runs unless I restart the flow and then it will run fine again. It seems to happen more often when the processor runs at a daily interval. My flows are using Nifi runtime 1.27.0.2.3.14.0-14 and I'm on CDP Public Cloud on Azure.```org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException: Failed to acq

Apache NiFi 2.6.0 upstream ships with Angular 19.2.14. Apache did not change the Angular dependency within the NiFi 2.6.x release line. Instead, the Angular dependency update was introduced as part of the NiFi 2.7.0 development cycle.In contrast, CFM modified the Angular dependency while continuing to ship NiFi 2.6.0.CFM 4.12.0.0 upgraded Angular from 19.2.14 to 19.2.19. After this change, the frontend no longer sent the Request-Token header with API requests, causing NiFi’s XSRF protection to reject legitimate UI operations. This regression was later addressed in CFM 4.12.0.1.However, instead of remaining on the Angular 19.x patch line, CFM 4.12.0.1 upgraded Angular again to 20.2.4.After inspecting the compiled frontend bundled with CFM 4.12.0.1, I confirmed that it is built with Angular 20.2.4. I also verified that the bundled frontend still follows the pre-fix implementation for GHSA-58c5-g7wp-6w37 and does not contain the upstream security fix. According to Angular’s official secur

I have been trying to send data from Kafka to snowflake using the JDBC driver with Kafka connect. some details about the environment:Kafka is running in a Cloudera private cluster (Base 7.1.9).The version of the Snowflake-kafka connector driver is 2.2.2.The configuration of the Kafka (v3.2.0+) connector is:{ "buffer.count.records": "2", "buffer.flush.time": "60", "buffer.size.bytes": "5000000", "connector.class": "com.snowflake.kafka.connector.SnowflakeSinkConnector", "errors.log.enable": "true", "errors.log.include.messages": "true", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "snowflake.database.name": "kafka_db", "snowflake.private.key": "REDACTED", "snowflake.schema.name": "kafka_schema", "snowflake.topic2table.map": "snowflake_kafka_poc:existing_table1", "snowflake.url.name": "https://REDACTED.snowflakecomputing.com", "snowflake.user.name": "kafka_connector_user_1", "tasks.max": "4", "topics": "snowflake_kafka_poc", "value.converter": "com.

Hi all,We've upgraded our base cluster (Cloudera Runtime 7.1.9) to CDS 3.5via the new add-on parcel — that part is working fine.Now we're trying to make Spark 3.5 available as a selectable RuntimeAddon inside Cloudera AI Workbench sessions, but we're stuck.Our setup:- Cloudera AI Workbench version: 2.0.44-h1-b4- CDP Private Cloud Data Services: 1.5.4-h10-b2- Deployment type: ECS- We use Spark Pushdown (sessions submit jobs that run on the basecluster via YARN)What we've found so far:- Currently only Spark versions up to 3.3.2 show up in SiteAdministration > Runtime > Runtime Addons- There's a SPARK3 3.5.4 parcel visible in Cloudera Manager underour CAI cluster, listed as "Available Remotely" — not yetdownloaded/activatedHas anyone gotten Spark 3.5 working as a Runtime Addon on a similarsetup? Specifically:1. Is downloading/activating that SPARK3 parcel the right approach,or does it need to be "linked" to the workbench some other way?2. Does this require a minimum Workbench versio