Q&A

Hello when I try to View Data Provenance of a processor then it shows multiple rows but then when I try to view its content then I get "Content is no longer available in the content repository"though the size in KB of the input and output claims are there. Also surprisingly the new entries content I can see but the same thing does not show the next day.

I am getting an error when using the ExcelReader to convert an Excel sheet to JSON. Everything works except for some dates. If I remove the Date Format setting on the controller, no issue, but get the date in milliseconds from 1/1/1970, which I do not want. The ExcelReader is failing on the date 05/01/2017. This is a valid MM/dd/yyyy date. The error I get is: ERROR015de330-01a0-1000-0000-00001cb4e04eAll NodesConvertRecord[id=015de330-01a0-1000-0000-00001cb4e04e] Failed to process FlowFile[filename=Test_File.xlsx]; will route to failure: org.apache.nifi.processor.exception.ProcessException: Could not parse incoming data- Caused by: org.apache.nifi.serialization.MalformedRecordException: Read next Record from Excel XLSX failed on row 59 in sheet Data- Caused by: org.apache.nifi.serialization.record.field.FieldConversionException: Conversion failed for [05/01/2017] named [column_9] to [java.time.LocalDate] [java.time.format.DateTimeParseException] Text '05/01/2017' could not be parse

Hello,I have a 3 nodes cluster environment running Nifi version 1.29.  Nifi logs are sync to Elastic Search; I noticed that GetSFTP is reporting Processing Failed multiple times.  It appears the processor recover on its own for example from January 1st to present reported 392 times events.   I'm trying to identify, why this is occurring as the log doesn't show much beside that message.    Any idea why this is occurring?Example of message:GetSFTP[id=633332e2-0ad4-17db-85d0-2ec76b7a49bc] Processing failedThanks!

HelloHow to read the body of the mail in nifi? I don't see any relevant processor for this.I see there is an "ExtractEmailAttachments" processor but will it read body reliably? We tried to test and see that it is able to make two flowfiles after executing. One is with a text/plain and has the email body present and another one is text/html having some html content having what we sent in mail as well.But due to its name being "ExtractEmailAttachment" and not something like ExtractEmailBody I am not sure.How to effectively and reliably read the mail body? In NiFi 1.28 we had Python (which was able to read the body of mail) but in NiFi 2.8 we don't have Python and able to see GroovyScript which is not allowing import of javax.mail type libraries which is required for writing the logic to fetch the email body.

Hi There, We are currently using Cloudera On-Prem (CDP 7.1.9SP1, CFM 2.1.7, CM 7.13.1), where monitoring of Spark jobs (such as memory utilization, CPU usage, active jobs, completed jobs, and related metrics) appears to be primarily available through the UI.We explored several approaches for retrieving Spark Structured Streaming and job execution metrics programmatically. However, most publicly available documentation references accessing the Spark Driver UI via port 4040 (same case with 4041,4042,4043 etc) on the driver node. In our Cloudera environment, this endpoint does not appear to be operational or accessible, which limits our ability to leverage those documented methods.Could you please point us in the right direction or suggest alternative approaches for retrieving such monitoring data through command-line interfaces, APIs, logs, event stores, or any other available mechanisms within Cloudera?At this stage, we are not looking for a specific metric alone. Our objective is

Hi all — we're migrating from NiFi Registry to the git-based Flow Registry Client, specifically the BitbucketFlowRegistryClient (NiFi 2.11.0), and validating the workflow.From what we've seen so far, committing a flow version via the NiFi UI pushes directly to the configured Bitbucket branch, with no built-in approval step — any review has to happen externally via a Bitbucket PR after the fact.Questions for the community:Approval workflow: Is there any NiFi-native mechanism (now or planned) to gate a version commit behind a review/approval, or is external git-host PR review the intended pattern?Branching: For teams enforcing PR-based review on flow changes, do you commit to a feature branch by default and require manual PR creation, or is there tooling to automate opening the PR after a NiFi commit?Diffing: Any recommended pattern for diffing flow JSON in PRs on Bitbucket specifically? We saw Datavolo's Flow Diff GitHub Action for GitHub — is there an equivalent for Bitbucket, or does

Hello.Before deleting rows in a specific table (463,462 rows in table), HDFS file size is :$ hadoop fs -du -s -h /apps/hive/warehouse/prd_thmil.db/th_mil_fb_code_value_brut54.2 M 162.6 M /apps/hive/warehouse/prd_thmil.db/th_mil_fb_code_value_brut54.2 Mb is the size of 1 file and each file is replicated 2 times so 162.6 Mb is the total size, it OK.But after deleting more than 450,000 rows in the table (12,890 rows remaining after the DELETE), the file size didn't change at all.Is it normal ? When new rows are added in the table, file size won't grow and HDFS will 'overwrite' older data with the new one ?Regards

I'm running several ETL flows in CDF/Nifi. The flows all involve writing some data to Iceberg tables. These flows are mostly running on a schedule that ranges from hourly to weekly (i.e., infrequent writes). Each data flow deployment has a few independent DAGs that all end with the PutIceberg processor, such that there are multiple separate ETL processes running at different intervals all in the same Nifi deployment.The problem I see occurs pretty rarely (~once a month) and only for some deployments. Occasionally, the PutIceberg processor will error with 'Failed to acquire a SAS token' (see full error log below).This will continue happening every time the processor runs unless I restart the flow and then it will run fine again. It seems to happen more often when the processor runs at a daily interval. My flows are using Nifi runtime 1.27.0.2.3.14.0-14 and I'm on CDP Public Cloud on Azure.```org.apache.hadoop.fs.azurebfs.contracts.exceptions.SASTokenProviderException: Failed to acq

Apache NiFi 2.6.0 upstream ships with Angular 19.2.14. Apache did not change the Angular dependency within the NiFi 2.6.x release line. Instead, the Angular dependency update was introduced as part of the NiFi 2.7.0 development cycle.In contrast, CFM modified the Angular dependency while continuing to ship NiFi 2.6.0.CFM 4.12.0.0 upgraded Angular from 19.2.14 to 19.2.19. After this change, the frontend no longer sent the Request-Token header with API requests, causing NiFi’s XSRF protection to reject legitimate UI operations. This regression was later addressed in CFM 4.12.0.1.However, instead of remaining on the Angular 19.x patch line, CFM 4.12.0.1 upgraded Angular again to 20.2.4.After inspecting the compiled frontend bundled with CFM 4.12.0.1, I confirmed that it is built with Angular 20.2.4. I also verified that the bundled frontend still follows the pre-fix implementation for GHSA-58c5-g7wp-6w37 and does not contain the upstream security fix. According to Angular’s official secur

I have been trying to send data from Kafka to snowflake using the JDBC driver with Kafka connect. some details about the environment:Kafka is running in a Cloudera private cluster (Base 7.1.9).The version of the Snowflake-kafka connector driver is 2.2.2.The configuration of the Kafka (v3.2.0+) connector is:{ "buffer.count.records": "2", "buffer.flush.time": "60", "buffer.size.bytes": "5000000", "connector.class": "com.snowflake.kafka.connector.SnowflakeSinkConnector", "errors.log.enable": "true", "errors.log.include.messages": "true", "key.converter": "org.apache.kafka.connect.storage.StringConverter", "snowflake.database.name": "kafka_db", "snowflake.private.key": "REDACTED", "snowflake.schema.name": "kafka_schema", "snowflake.topic2table.map": "snowflake_kafka_poc:existing_table1", "snowflake.url.name": "https://REDACTED.snowflakecomputing.com", "snowflake.user.name": "kafka_connector_user_1", "tasks.max": "4", "topics": "snowflake_kafka_poc", "value.converter": "com.

Hi all,We've upgraded our base cluster (Cloudera Runtime 7.1.9) to CDS 3.5via the new add-on parcel — that part is working fine.Now we're trying to make Spark 3.5 available as a selectable RuntimeAddon inside Cloudera AI Workbench sessions, but we're stuck.Our setup:- Cloudera AI Workbench version: 2.0.44-h1-b4- CDP Private Cloud Data Services: 1.5.4-h10-b2- Deployment type: ECS- We use Spark Pushdown (sessions submit jobs that run on the basecluster via YARN)What we've found so far:- Currently only Spark versions up to 3.3.2 show up in SiteAdministration > Runtime > Runtime Addons- There's a SPARK3 3.5.4 parcel visible in Cloudera Manager underour CAI cluster, listed as "Available Remotely" — not yetdownloaded/activatedHas anyone gotten Spark 3.5 working as a Runtime Addon on a similarsetup? Specifically:1. Is downloading/activating that SPARK3 parcel the right approach,or does it need to be "linked" to the workbench some other way?2. Does this require a minimum Workbench versio

PVC 서비스 재시작 후 'Pod Count' 서비스 재시작이전보다 감소하는 증상이 발생하여 정상유무를 확인하고자문의를 드립니다.+ 첨부 그림 파일처럼 서비스 재시작전에는 "Pod Cont" 갯수가 54개로 표시되지만, 서비스 재시작 후 18개로 줄어들어서 해당 증상이 비정상일 경우 확인해야 되는 항목과 참고할 수 있는 로그정보가 필요합니다.'Pod Count'에는 어떤 정보가 포함되어 있나요? 

Hi, Recently in one of our critical Cloudera CDP 7.1.7_p1044 cluster we could see that Impala is getting hanged intermittently and whenever Impala session are getting hanged no one was being able to run even Show databases and all Impala sessions are getting hanged. While restarting the Impala Catalog service the issue gets resolved but again it reappears after 5-6 hours or may be after 1 days as there is no fixed time.We checked the Impala catalog log and found the following errors when it is in hung state: "Error getting policies; Received NULL response!!. secure Mode=true, user=impala/pc1133.aaa.abc.com@corp.xxx.COM (auth: KERBEROS), service Name=hive" Also while checking the Ranger errors we got the following error : "RangerAdminRESTClient.java:167] Error getting policies; Received NULL response!!" Also getting error in Hive Server 2 logs: "Error occurred during processing of message.SSL Handshake Exception: Remote host terminated the handshake" A

I have copied hive table from Prod to QA by using get and put command but while executing select counnt (*) is it showing 0 count but i can see physical location is having data however i have checked all permission are in place so i did invalidate metadata and table refresh and msck repiare but still facing same issue. can seomeone help. RegardsSaim

Hello TeamWe have an increasing number of applications being added to our NiFi server. Now the nifi has got too slow that it won't load the home page or when we click a process group then it won't open despite browser url showing the process group id as if page should have gone there.Also the apis we have implemented using HandleHttpRequest return Socket Timeout error.Please note these were working fine earlier. What should we do?

I have set my impala.admission-control.min-query-mem-limit.root.default to 2GiB and my impala.admission-control.max-query-mem-limit.root.default to 57GiB, which aligned with the memory allocation for each Impala Executor node that I've set on the resource templates. My problem is, when I'm running a heavy query (I tried to set it so the limit is 10000000), the limit memory is set to 2GiB and makes the query OOM, which is strange because the max-query-mem-limit is set to 57GiB, so the limit should be > 2GiB and <= 57GiB if the query is heavy, right? Yet it's just set to the minimum mem limit. I know that the limit is set to clamp(min memory limit per query, memory estimated, max memory limit per query), but the estimated memory is 4GB and what makes it stranger is that when I try to define SET MEM_LIMIT in the query to another number like 4G or 8G, the estimated memory changes; which means there's some calculation for the estimated memory based on the memory li

I missed my CDP-0011 certification appointment. What should I do?The certification fee is very expensive for me, so I cannot repurchase it. Has this situation happened to any of you, and what is the correct action I should take?

 Hi all,I am working with Cloudera Data Visualization 7.1.6 and creating a dashboard that includes a KPI visual.The visual uses a dataset based on a Hive query. Is there any way to pass dashboard filter values or parameters directly to the dataset query?Based on the official documentation and the information I found online, it appears that Cloudera Data Visualization applies filters only at the visualization or UI level and does not support passing parameters to the underlying dataset query.Could someone confirm whether this is an architectural limitation, or whether there is an alternative solution or workaround?Thanks in advance for any suggestions or help.

HiWe are having problems with a single instance of nifi 2.5.0 on a physical windows 11 machine with 64 CPU cores, 64 GB RAM and NVME 12 TB ( 4 x 3 TB drives ) striped D : drive set ingesting slowly.  C: drive is for windows.I was hoping someone could maybe give a sequential list of things to check so we at least are approaching this the right way.The people who set up the machine have put the ingest directory ( ingest files via Getfiles ), and the nifi instance ( content repo and everything else ) on the same single D ; drive striped disk set.  Being a striped set I wouldn't have thought speed would be an issue even with both ingest and content repo on the same striped set. So one thing I wondered was can we force nifi to use all cores, or do we need keep some cores  ( maybe 4 ? ) for windows to use? And if we did use say 60 cores, what is the best way to do this -  and is it set at a nifi processor level,  or do you need to force Nifi to use all core

I think I might just be misunderstanding something, because it seems like NIFI requires HTTPS for intra cluster communication and uses the SAME HTTPS channel for external UI and API access?I'm expecting to have intra cluster communication be separate from my load balancer/ingress. Meaning the nodes communicate securely with each other but I can set the API and UI to HTTP so I can terminate that at my load balancer- this is very standard practice.What I think I'm seeing is NIFI uses the same HTTPS channel for both internal and external communication? That can't be right because there is no way I'm registering all my NIFI nodes with a public DNS domain.But this can't be the case right? I should be able to set just the end user API and UI to HTTP and use a self signed cert for intra-node communication.I can't even seem to be able to just disable HTTPS entirely, nifi's config is poorly documented and disabling HTTPS seems to break everything. I have this on a secured cluster and because th

 I have a question regarding the long-term support alignment for CDP Private Cloud Base 7.1.9.As you know, CDP 7.1.9 is an LTS release supported until October 2028. Initially, this version only supported Python 3.8, but due to the relatively early EOL of Python 3.8, support for Python 3.11 was introduced starting from SP1 CHF9.On the other hand, CDP 7.1.9 depends on OpenJDK 17, whose end of support is currently December 31, 2027. This creates a potential misalignment with the CDP LTS support period extending into late 2028.Given the precedent with Python (i.e., adding support for a newer version within the same LTS release via CHF updates), we would like to understand the roadmap for Java:Are there any plans to support a newer OpenJDK version (e.g., OpenJDK 21 or later) within CDP 7.1.9 through future SP/CHF updates?If so, is there an estimated timeline or target CHF/SP where this support might be introduced?If not, how does Cloudera plan to address the gap between the OpenJDK 17

We are running a sparkly job that runs queries on cloudera CDP hive cluster. The job sometimes stops before a dbwriteTable function, doing nothing and running indefinitely. The job doesn't always stop in the same point, but always during this trywrite function invocation, not catching any error:trywrite = function(sc, new_name, df, log_obj, wait_sec = 600, max_wait = 3600) { start_time = Sys.time() while (difftime(Sys.time(), start_time, units = 'secs') <= max_wait) { print(paste0('Attempt to write table: ', new_name, ' - ', Sys.time())) # Connection is valid? if (!DBI::dbIsValid(sc)) { error(log_obj, paste0('Connection not valid during write table: ', new_name)) stop(paste0('Failed to write table: ', new_name)) } tryCatch({ print(paste0('Writing table: ', new_name)) result = DBI::dbWriteTable(sc, new_name, df) print(paste0('Write completed table: ', new_name, ' - ', Sys.time())) return(result) }, error = function(

During my preparation, I used multiple learning resources, including official Cloudera documentation, hands-on practice labs, and practice questions, which helped me assess my knowledge and identify areas that needed improvement. Some tips for candidates preparing for the exam: Focus on understanding concepts rather than memorization.Gain hands-on experience with Cloudera technologies.Review the official exam objectives carefully.Take practice tests to evaluate your readiness.Spend extra time on troubleshooting and real-world scenarios. I would like to thank the Cloudera community for the valuable discussions and learning resources that contributed to my success. Best wishes to everyone currently preparing for their Cloudera certification exams. Keep learning and stay focused—you can do it!

 **Environment**- Cloudera CFM 2.1.7 / Apache NiFi 1.26.0- Cloudera CDP 7.3.1- Flow source: Cisco ASA Firewall (IPFIX export) **Issue** The `ListenNetFlow` processor successfully decodes standard IPFIX/NetFlow v9 fields but produces a WARN for every packet that contains Cisco ASA enterprise fields: ```WARN com.cloudera.nifi.netflow.ListenNetFlow: Sender [/x.x.x.x] Field Type [33002] Length [2] Field Decoder not foundWARN com.cloudera.nifi.netflow.ListenNetFlow: Sender [/x.x.x.x] Field Type [33001] Length [12] Field Decoder not foundWARN com.cloudera.nifi.netflow.ListenNetFlow: Sender [/x.x.x.x] Field Type [33000] Length [12] Field Decoder not foundWARN com.cloudera.nifi.netflow.ListenNetFlow: Sender [/x.x.x.x] Field Type [40000] Length [12] Field Decoder not found ``` At production NetFlow volume, this generates thousands of WARN lines per second, making `nifi-app.log` effectively unusable for monitoring other issues. More critically, the data in thes

"Hello everyone, I am researching the licensing model for Cloudera Private Cloud. Could someone clarify the baseline metrics for an On-Premises deployment? Is the subscription billed per Node or per CPU Core? Also, is there a minimum number of nodes required for a starting enterprise subscription? We want to budget this for our upcoming infrastructure planning."