Q&A

I am working with Apache NiFi and trying to replicate a pattern that we had done in existing system.In existing system, we have multiple (around 10) email accounts, and we iterate over them using a loop, connecting to each mailbox, reading emails, and processing them.In NiFi, I have configured a ConsumePOP3 processor and it works fine for a single email account. However, I need to handle multiple email accounts in a scalable and maintainable way. We are not able to connect any processor or something before ConsumePOP3.

HiIam using ListSMB which lists all the files correctly on the source system.However, not all files are fetched using FetchSMB, some of the files are skipped with below error.could this be due to // in the file path ?is it due to some issue on the SMB server settings ?FetchSmb[id=03ab6b24-019d-1000-0000-000032e5cf1d] Could not fetch file SDM-Prod-Reports/EOS_TFC_Metrics//EOS_TFC_Metrics.csv.: java.io.IOException: Could not create session for share smb://ifiler-smb03:445/SFTP- Caused by: com.hierynomus.smbj.common.SMBRuntimeException: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired- Caused by: com.hierynomus.protocol.transport.TransportException: java.util.concurrent.ExecutionException: com.hierynomus.smbj.common.SMBRuntimeException: java.util.concurrent.TimeoutException: Timeout expired- Caused by: java.util.concurrent.ExecutionExcep

Hello everyone,I’m currently facing a performance issue with Cloudera Impala when using it through the ODBC driver.Here is my environment:CDH: 6.2Impala: 3.2.0 (Impala 2.7.1 ODBC Driver)Integration: SAS 9.4 M7 (SAS Access to Impala)Authentication: Active Directory (user/mdp)Security: SSL enabledI have configured my /etc/odbc.ini as follows:[Impala_KRB14]Description=ODBC Cloudera Impala SSL Node 14Driver=/opt/cloudera/impalaodbc/lib/64/libclouderaimpalaodbc64.soHOST=""""""""" ##hiddenPORT=21050Database=""""""""""" ##hiddenAuthMech=3SSL=1TrustedCerts=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pemLogLevel=2LogPath=/tmp/impala_odbc.logSSP_BATCH_SIZE=60536RowsFetchedPerBlock=60536TSaslTransportBufSize=4000I also tried adding several ODBC options/tuning as seen in the odbc.ini parameters, but unfortunately I did not see any performance improvement.The issue is mainly that:Opening a table through SAS takes a long timeEven simple queries like SELECT ... WHERE ... are very slowI would like t

I have an old cluster that had some hard drives fail. On one server (node06), the OS drive failed and instead of building it from scratch, I created an image from another node (node02) in the cluster. I then started the new server (node06) and changed the name and the IP address to match node06. But when I put it back on the network, Cloudera kind of went crazy because it thought that node was node02 and duplicated it. Is there something I need to do on the imaged node (node06) so it doesn't think it is node02? I hope this makes sense.

Environment - NiFi 1.23.2, 3-node clusterProblemAll of sudden, all the cluster nodes becomes unavailable:Spoiler (Highlight to read)Error: Received disconnection request message from cluster coordinator with explanation: org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption.Disconnecting node due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption.Event Reported for atl-fm-nifi01.corp.pps.io:8443 -- Node disconnected due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption.Erro

Hi allI am using Knox for CDP SSO and would like to Enable the REST Catalog Service but use Keycloak for SSO. Is this possible? Private CDP, not Cloud. Thanks

Hi, Nifi keep disconnecting due to below error , appreciated any help .Node disconnected from cluster due to org.apache.nifi.controller.serialization.FlowSynchronizationException: Failed to connect node to cluster because local flow controller partially updated. Administrator should disconnect node and review flow for corruption.2023-09-21 12:34:09,572 INFO [main] o.a.n.c.l.e.CuratorLeaderElectionManager Cannot unregister Leader Election Role 'Primary Node' because that role is not registered

Dears,The Hive MERGE is failing with the error below. Since yearmonth is a partition column, implemented a Type-2 SCD script to track changes. However, even after including the partition column in the join condition, the same error persists.Error while compiling statement: FAILED: SemanticException [Error 10007]: Ambiguous column reference deal_yearnumber in tgt.ScriptMERGE INTO accrued_interest_metric tgtUSING temp_ACCRUED_INTEREST_METRIC srcON tgt.deal_number = src.deal_numberAND tgt.deal_metric_end_date = DATE '9999-12-31'WHEN MATCHEDAND tgt.deal_hash <> src.deal_hashTHEN UPDATE SETdeal_metric_end_date = CURRENT_DATE - 1,update_date = current_timestamp()WHEN NOT MATCHED THENINSERT (deal_number,deal_metric_start_date,deal_metric_end_date,deal_branch_cd,deal_type_cd,deal_reference_number,currency_cd,deal_maturity_date,deal_last_rollover_date,deal_next_rollover_date,deal_term_type_cd,deal_accrued_interest_rate,status_cd,status_description,deal_account_number,deal_interset_account

Edit: Tried using CSVWriter, XMLWriter and AvroWriter. All NG. Nifi Version2.5.0ProcessorQueryDatabaseTableRecord Drivercom.microsoft.sqlserver.jdbc.SQLServerDriver I have a requirement to pull varbinary data from a table however the  output is always text.  I expect a result starting with 0x. According to the jdbc driver within nifi, varbinary is supported however I see it gets converted to bytes. Is there a way to maintain the 0x format for the writer? NotesJDBC source for statement varbinary is supported at https://github.com/apache/nifi/blob/main/nifi-extension-bundles/nifi-extension-utils/nifi-database-utils/src/main/java/org/apache/nifi/util/db/JdbcCommon.java (line 360)

Hello All,i have plan to renew the tls certificates for my prod cluster. I have already generated csr for all hosts and got the signed from my clients. Could you please share the steps for renewing. I gone through cloudera documentation but still having doubts. Can someone help. It would be much appreciated. 

Hello, Does NiFi’s PutS3Object processor perform an MD5 integrity check for the files uploaded to Amazon S3?Thank youCristian

Hello everyone, I have been trying to solve an issue on my on-prem NiFi 2.4.0 application for a month which being PutEmail timeout errors. Specifically, we were using NiFi 1.9.0 and local smtp server to send mails, after updating to 2.4.0 we started using office365 smtp server. swaks --server smtp.office365.com --port 587 --tls --helo test.local --auth LOGIN --auth-user notifymail@****.com --auth-password '****' --from notifymail@****.com --to okan.ergun@****.com --header "Subject: Test Email" --body "Test SMTP Relay Service"With the command above I never had timeouts, and every mail was successful from the application environment, although NiFi gets timeouts time to time and doesn't have any specific times to have timeouts. Processor Details:SMTP Hostnamesmtp.office365.comSMTP Port587Authorization ModeUse PasswordSMTP Usernamenotifymail@****.comSMTP PasswordSensitive value setSMTP AuthtrueSMTP STARTTLStrueSMTP Socket Factory${literal('')}SMTP X-Mailer HeaderNiFiAttribut

I am a Korean user. Recently, I received instructions to address the Apache Shiro security issue pointed out by the Korean Financial Supervisory Service. I am currently using CDH 7.1.8-1 and was instructed to update Shiro from the current version 1.11 to 1.13 or higher. As far as I know, it is being used in multiple places such as Knox and Kafka. Could you please let me know how to update it as soon as possible?

production pyspark jobs failure, on prem cloudera version 7.1.8 not supported at present. ranger kms health is unhealthy and failing pyspark job write to lake any suggestion to resolve ranger kms server issues. 

I'm running into an issue with custom runtime addon uploads on CML 2.0.44-h1-b4 (Kubernetes). Wondering if anyone else has seen this or can confirm my fix is safe.**Problem:**When uploading a runtime addon via the API endpoint `/api/v2/runtimeaddons/custom`, it shows "Initialized" in the UI, then fails after ~5 minutes with no error message. API logs show:```ERROR api.runtimeaddons Finish writing Custom Runtime Addon: failed.ERROR api.runtimeaddons Finish adding new custom runtime addon failed: Failed saving addon content to NFS.```**What I Found:**After digging through the deployment configs, the API pod is missing the `projects-pvc` volume mount. Other pods like `ds-vfs` and `s2i-client` have it mounted at `/projects`, but the API deployment doesn't.```bash# PVC shows it's used by ds-vfs and s2i-client, but NOT apikubectl describe pvc projects-pvc# API deployment has no projects-pvc volumekubectl get deploy/api -o yaml | grep projects# (returns nothing)```**Proposed Fix:**Add the vol

I am trying to setup LDAP in NiFi (v2.5.0) , and run into issues that it doesnt show all users from sub-ous,User Search Base: ou=Users,dc=example,dc=com, and User Search Scope is SUBTREEIt shows some users, but not all.If I change the User search base to the exact sub OU, and then it will show the expected users, e.g.  ou=XYZ,ou=ABC,ou=Users,dc=example,dc=comI thought by specifying the user search base at a higher level, with substree as the search scope, it should list all users below?Can someone advise anything else I should be changing in the LDAP User/Group provider settings to get this working?

 We are working with Apache NiFi version 1.28 and need to validate JWT tokens that arrive as HTTP headers in FlowFiles created by the HandleHttpRequest processor. Once validated, the flow should continue to process the request; if invalid or expired, the request should be rejected.I am aware that scripting options exist (e.g., ExecuteScript with Python/Groovy and JWT libraries), but I would like to know if there is a recommended or supported best practice within NiFi for handling JWT validation in this scenario.Our use case is:Validate the JWT signature and claims against the JWKS endpoint provided by the identity provider.Extract claims for routing/authorization decisions.Reject invalid or expired tokens before further processing.Any guidance on the best way to implement this securely and efficiently would be greatly appreciated.@MattWho 

Hi everyone,I'm currently trying to write records containing timestamps to parquet.The writing part seems to work fine, as the resulting parquet file contains the avro schema '{"name":"ts_string","type":{"type":"long","logicalType":"timestamp-millis"}}' and the parquet column type "timestamp[ms, tz=UTC]".However when reading the same parquet file, I get the error message:ERROR [Timer-Driven Process Thread-6] o.a.n.processors.standard.ConvertRecord ConvertRecord[id=c2746584-b622-3127-f4fd-c46a9243553e] Failed to process StandardFlowFileRecord[uuid=cef04323-a7c0-4965-84d6-c2b8d16c3fac,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1768470437932-1, container=default, section=1], offset=17423, length=6057],offset=0,name=73ca82fa-02af-44d1-9c23-27e0b2b6588f,size=6057]; will route to failurejava.lang.ClassCastException: class java.time.Instant cannot be cast to class java.lang.Long (java.time.Instant and java.lang.Long are in module java.base of loader 'bootstrap')at org.

I'm experiencing very slow HDFS start in CDP 7.1.7SP1 for a cluster with a huge number of blocks (over 300 million, with each server having up to 40 million)I've checked thishttps://community.cloudera.com/t5/Community-Articles/Scaling-the-HDFS-NameNode-part-5/ta-p/327450and I wonder if setting dfs.blockreport.split.threshold to 0 might somehow speed up the processI've seen that the setting should go in NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xmIs this setting service wide so that a full restart is necessary?

I'd like to add runtimecatalog on the CML service. I'm currently using 12.8cuda version and Nvidia H100Gpu, and I'd like to use 3.12 jupyterlab. Is this version supported? When I looked it up on github, there didn't seem to be anything. Or can I customize it and use it after image build? Please adviseThank you.

Hi Team,We are using NIfi 1.26 in the Dev/Prod and we would like to get rid of using the Nifi template usage instead we are intend to use Flow definitions. However the users are still using the import and export method. Is there a way to disable the operations of import and export templates for all the users. 

Hi Team,I'm trying to flatten a nested JSON and it is working fine for few cases which has data but for some cases if it has [] empty then the record is not showing up in the result. Could you please help me in tackling this corner case.My data:{ "data": { "getVarietyListing": { "edges": [ { "node": { "id": "346595", "identifier": 503882, "description": "Estefania (PL)", "productLifecycleManagementAreas": [] } }, { "node": { "id": "351741", "identifier": 504382, "description": "Amsterdam 2 - ABK", "productLifecycleManagementAreas": [ { "code": { "code": "X00", "description": "Prepare to retire" }, "startDate": "2025-09-01", "futureStartDate": null, "futureCode": null, "area": { "country": "EA

Hi everyone, I hope you’re doing well. I am working on a dataflow in Apache NiFi 1.18, and I need to retrieve the queue size information (flowfile count and content size) directly within NiFi itself, not via an external script or Postman. I know that the NiFi REST API provides this data, and I can access it successfully using external tools. However, my goal is to access queue metrics from inside NiFi, for example through processors like InvokeScriptedProcessor, QueryNiFiReportingTask, or any other built-in mechanism, without sending an external REST API request from outside NiFi. Is there a recommended approach, processor, or reporting task that allows NiFi to read its own queue sizes internally? If not, what would be the best practice to achieve this? Any guidance or examples would be greatly appreciated. Thank you in advance!    

I launch nifi server using docker with connection to external Keycloak:nifi: hostname: <IP> image: apache/nifi:latest environment: AUTH: oidc NIFI_SECURITY_USER_OIDC_DISCOVERY_URL: https://<keycloak_url>/realms/<realm>/.well-known/openid-configuration NIFI_SECURITY_USER_OIDC_CLIENT_ID: <client_id> NIFI_SECURITY_USER_OIDC_CLIENT_SECRET: <client_secret> NIFI_SECURITY_USER_OIDC_ADDITIONAL_SCOPES: email NIFI_SECURITY_USER_OIDC_CLAIM_IDENTIFYING_USER: preferred_username NIFI_SECURITY_USER_OIDC_TRUSTSTORE_STRATEGY: NIFI NIFI_CLUSTER_NODE_PROTOCOL_PORT: 6007 NIFI_SENSITIVE_PROPS_KEY: <key> SINGLE_USER_CREDENTIALS_USERNAME: admin SINGLE_USER_CREDENTIALS_PASSWORD: <pass> NIFI_WEB_HTTPS_HOST: <IP> NIFI_WEB_PROXY_HOST: <domain> NIFI_WEB_HTTPS_PORT: 8443 KEYSTORE_PATH: /opt/certs/keystore.p12 KEYSTORE_TYPE: PKCS12 KEYSTORE_PASSWORD: <pass> TRUSTSTORE_PATH: /opt/certs/truststore.p12

I am trying to connect between NiFi and NiFi Registry. I'm able to authenticate with Registry from NiFi but I'm unable to see the buckets, not even the public ones. I've configured a SSL Context on NiFi that references a truststore containing Registry's Truststore and Keystore. I've imported NiFi's node certificates into the truststore on the registry's side. I've given the 'proxy and manage user bucket' permissions to the Node's identity in the Registry UI and Read Write and Execute permission to the same user on the Buckets. But I'm still unable to see the buckets on NiFi's UI. The API Responses also indicate that NiFi only has read permissions to the buckets. This is as if there is some anomaly during login.. Note:1) I'm using a Clustered NiFi Setup. I have verified that my node identity is 'CN=node-0-nifikop'.2) Both NiFi Cluster (NiFiKop) and the NiFi Registry (Helm Release) are running inside a Kubernetes ClusterError Samples:The Error on NiFi UI:  Error.jpegRegistry Bu