Ask the Community

I am a Korean user. Recently, I received instructions to address the Apache Shiro security issue pointed out by the Korean Financial Supervisory Service. I am currently using CDH 7.1.8-1 and was instructed to update Shiro from the current version 1.11 to 1.13 or higher. As far as I know, it is being used in multiple places such as Knox and Kafka. Could you please let me know how to update it as soon as possible?

production pyspark jobs failure, on prem cloudera version 7.1.8 not supported at present. ranger kms health is unhealthy and failing pyspark job write to lake any suggestion to resolve ranger kms server issues. 

I'm running into an issue with custom runtime addon uploads on CML 2.0.44-h1-b4 (Kubernetes). Wondering if anyone else has seen this or can confirm my fix is safe.**Problem:**When uploading a runtime addon via the API endpoint `/api/v2/runtimeaddons/custom`, it shows "Initialized" in the UI, then fails after ~5 minutes with no error message. API logs show:```ERROR api.runtimeaddons Finish writing Custom Runtime Addon: failed.ERROR api.runtimeaddons Finish adding new custom runtime addon failed: Failed saving addon content to NFS.```**What I Found:**After digging through the deployment configs, the API pod is missing the `projects-pvc` volume mount. Other pods like `ds-vfs` and `s2i-client` have it mounted at `/projects`, but the API deployment doesn't.```bash# PVC shows it's used by ds-vfs and s2i-client, but NOT apikubectl describe pvc projects-pvc# API deployment has no projects-pvc volumekubectl get deploy/api -o yaml | grep projects# (returns nothing)```**Proposed Fix:**Add the vol

I am trying to setup LDAP in NiFi (v2.5.0) , and run into issues that it doesnt show all users from sub-ous,User Search Base: ou=Users,dc=example,dc=com, and User Search Scope is SUBTREEIt shows some users, but not all.If I change the User search base to the exact sub OU, and then it will show the expected users, e.g.  ou=XYZ,ou=ABC,ou=Users,dc=example,dc=comI thought by specifying the user search base at a higher level, with substree as the search scope, it should list all users below?Can someone advise anything else I should be changing in the LDAP User/Group provider settings to get this working?

 We are working with Apache NiFi version 1.28 and need to validate JWT tokens that arrive as HTTP headers in FlowFiles created by the HandleHttpRequest processor. Once validated, the flow should continue to process the request; if invalid or expired, the request should be rejected.I am aware that scripting options exist (e.g., ExecuteScript with Python/Groovy and JWT libraries), but I would like to know if there is a recommended or supported best practice within NiFi for handling JWT validation in this scenario.Our use case is:Validate the JWT signature and claims against the JWKS endpoint provided by the identity provider.Extract claims for routing/authorization decisions.Reject invalid or expired tokens before further processing.Any guidance on the best way to implement this securely and efficiently would be greatly appreciated.@MattWho 

Hi everyone,I'm currently trying to write records containing timestamps to parquet.The writing part seems to work fine, as the resulting parquet file contains the avro schema '{"name":"ts_string","type":{"type":"long","logicalType":"timestamp-millis"}}' and the parquet column type "timestamp[ms, tz=UTC]".However when reading the same parquet file, I get the error message:ERROR [Timer-Driven Process Thread-6] o.a.n.processors.standard.ConvertRecord ConvertRecord[id=c2746584-b622-3127-f4fd-c46a9243553e] Failed to process StandardFlowFileRecord[uuid=cef04323-a7c0-4965-84d6-c2b8d16c3fac,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1768470437932-1, container=default, section=1], offset=17423, length=6057],offset=0,name=73ca82fa-02af-44d1-9c23-27e0b2b6588f,size=6057]; will route to failurejava.lang.ClassCastException: class java.time.Instant cannot be cast to class java.lang.Long (java.time.Instant and java.lang.Long are in module java.base of loader 'bootstrap')at org.

I'm experiencing very slow HDFS start in CDP 7.1.7SP1 for a cluster with a huge number of blocks (over 300 million, with each server having up to 40 million)I've checked thishttps://community.cloudera.com/t5/Community-Articles/Scaling-the-HDFS-NameNode-part-5/ta-p/327450and I wonder if setting dfs.blockreport.split.threshold to 0 might somehow speed up the processI've seen that the setting should go in NameNode Advanced Configuration Snippet (Safety Valve) for hdfs-site.xmIs this setting service wide so that a full restart is necessary?

I'd like to add runtimecatalog on the CML service. I'm currently using 12.8cuda version and Nvidia H100Gpu, and I'd like to use 3.12 jupyterlab. Is this version supported? When I looked it up on github, there didn't seem to be anything. Or can I customize it and use it after image build? Please adviseThank you.

Hi Team,We are using NIfi 1.26 in the Dev/Prod and we would like to get rid of using the Nifi template usage instead we are intend to use Flow definitions. However the users are still using the import and export method. Is there a way to disable the operations of import and export templates for all the users. 

Hi Team,I'm trying to flatten a nested JSON and it is working fine for few cases which has data but for some cases if it has [] empty then the record is not showing up in the result. Could you please help me in tackling this corner case.My data:{ "data": { "getVarietyListing": { "edges": [ { "node": { "id": "346595", "identifier": 503882, "description": "Estefania (PL)", "productLifecycleManagementAreas": [] } }, { "node": { "id": "351741", "identifier": 504382, "description": "Amsterdam 2 - ABK", "productLifecycleManagementAreas": [ { "code": { "code": "X00", "description": "Prepare to retire" }, "startDate": "2025-09-01", "futureStartDate": null, "futureCode": null, "area": { "country": "EA

Hi everyone, I hope you’re doing well. I am working on a dataflow in Apache NiFi 1.18, and I need to retrieve the queue size information (flowfile count and content size) directly within NiFi itself, not via an external script or Postman. I know that the NiFi REST API provides this data, and I can access it successfully using external tools. However, my goal is to access queue metrics from inside NiFi, for example through processors like InvokeScriptedProcessor, QueryNiFiReportingTask, or any other built-in mechanism, without sending an external REST API request from outside NiFi. Is there a recommended approach, processor, or reporting task that allows NiFi to read its own queue sizes internally? If not, what would be the best practice to achieve this? Any guidance or examples would be greatly appreciated. Thank you in advance!    

I launch nifi server using docker with connection to external Keycloak:nifi: hostname: <IP> image: apache/nifi:latest environment: AUTH: oidc NIFI_SECURITY_USER_OIDC_DISCOVERY_URL: https://<keycloak_url>/realms/<realm>/.well-known/openid-configuration NIFI_SECURITY_USER_OIDC_CLIENT_ID: <client_id> NIFI_SECURITY_USER_OIDC_CLIENT_SECRET: <client_secret> NIFI_SECURITY_USER_OIDC_ADDITIONAL_SCOPES: email NIFI_SECURITY_USER_OIDC_CLAIM_IDENTIFYING_USER: preferred_username NIFI_SECURITY_USER_OIDC_TRUSTSTORE_STRATEGY: NIFI NIFI_CLUSTER_NODE_PROTOCOL_PORT: 6007 NIFI_SENSITIVE_PROPS_KEY: <key> SINGLE_USER_CREDENTIALS_USERNAME: admin SINGLE_USER_CREDENTIALS_PASSWORD: <pass> NIFI_WEB_HTTPS_HOST: <IP> NIFI_WEB_PROXY_HOST: <domain> NIFI_WEB_HTTPS_PORT: 8443 KEYSTORE_PATH: /opt/certs/keystore.p12 KEYSTORE_TYPE: PKCS12 KEYSTORE_PASSWORD: <pass> TRUSTSTORE_PATH: /opt/certs/truststore.p12

I am trying to connect between NiFi and NiFi Registry. I'm able to authenticate with Registry from NiFi but I'm unable to see the buckets, not even the public ones. I've configured a SSL Context on NiFi that references a truststore containing Registry's Truststore and Keystore. I've imported NiFi's node certificates into the truststore on the registry's side. I've given the 'proxy and manage user bucket' permissions to the Node's identity in the Registry UI and Read Write and Execute permission to the same user on the Buckets. But I'm still unable to see the buckets on NiFi's UI. The API Responses also indicate that NiFi only has read permissions to the buckets. This is as if there is some anomaly during login.. Note:1) I'm using a Clustered NiFi Setup. I have verified that my node identity is 'CN=node-0-nifikop'.2) Both NiFi Cluster (NiFiKop) and the NiFi Registry (Helm Release) are running inside a Kubernetes ClusterError Samples:The Error on NiFi UI:  Error.jpegRegistry Bu

Dear Community,   After a successful installion of Cloudera Runtime 7.0.3, we tried to do a Kerberization process. (We did the same before with 5.14) Everything went fine with the Kerberos wizard, but after in the config deployment phase, the YARN NodeManagers failed to start with the following error:   Error starting NodeManagerorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize container executor at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:394) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:936) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:1016)Caused by: java.io.IOException: Linux container executor not configured properly (error=-1) at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.init(LinuxContainerExecutor.java:307) at org.apac

Hello!I'm newbe in NiFi and Groovy.I found here – "Solved: Re: Get the processor-group name in NIFI flow - Cloudera Community - 213662" (2018) – very interestion solution! Thank @mburgess !!!  I have related question.I want to build with Groovy dictionary for every proceccor where key = Proccesor ID, value = "bread crumbs" (full path to processor from root canvas – the string like this "PG1_Name" / "PG2_(Child_of_PG1)_Name" / "PG3_(Child_of_PG2)_Name" .The quesions are:How to get list of Process Groups in the root and in Each PG?How to get list of Processors in Each PG?How to organize in Groovy cycles for scan PGs?How to save / read data in/from Groovy dictionary /  NiFi cache?Thanks in advance for any help and suggestions!   

Dear Support Team,I’m reaching out to request an update to the email address linked to my Cloudera account.I have already reached to the Cloudera community but haven’t received a response yet, so I’m turning to you for assistance. Thank you in advance for your help .

Hello Folks,I’ve the issue with JoinEnrichment ProcessorThe ORIGINAL flowfile looks like below:{"ticketserialnumber" : "TSN00000012","tranheaderid" : "TH00000012","entrymethodid" : "EM6","deviceid" : "DEV22","prodid" : 1,"isbetrejectedbytrader" : true,"isexchangeticket" : false,"terdisplayid" : "TER41","createddate" : "2025-06-17 22:24:00.0","requestid" : "REQ00000012","userdisplayid" : "USR863","cartid" : "CART00000012","transactiontype" : 1}, {"ticketserialnumber" : "TSN0000001","tranheaderid" : "TH001","entrymethodid" : "EM001","deviceid" : "DEV001","prodid" : 1,"isbetrejectedbytrader" : false,"isexchangeticket" : false,"terdisplayid" : "TERM001","createddate" : "2025-06-17 22:10:00.0","requestid" : "REQ001","userdisplayid" : "USER001","cartid" : "CART001","transactiontype" : 1},And the ENRICHMENT Flowfile is looks like below:{"tsn" : "TSN00000008","uuid_schema" : "VREQ00000005","event_date" : "2025-06-17 22:26:53.610071","terminalid" : "TER25","userid" : "USR244","locationid" : "LO

We are upgrading from 2.0.0 M4 to 2.4.0 in EKS via helm charts.But encountering issue while running nifi.sh file. Nifi.sh file references nifi-env.sh which is empty. We are suspecting issue is with docker image of nifi.When we opened binary of nifi 2.4.0 which is downloaded locally from internet, I can see content inside nifi-env.sh. Same is not present inside EKS when we deploy nifi 2.4.0 docker image into EKSThis is crashing our main container. Basically its not setting PID path.

Hello I'm trying to use Nifi Hashi Corp Vault Parameter Provider with Kubernetes authentication. - Nifi versions is 2.4.0 - Hashi Corp version is 1.19.5From my Nifi pod I get to authenticate successfully using the following curl curl -s -k -X POST -H 'X-Vault-Namespace: DATA/DATAOPE/EC002I002621/' -H 'X-Vault-Request: true' -d '{"jwt": "<content of file /var/run/secrets/kubernetes.io/serviceaccount/token>", "role": "role_k8s_auth_demo_re_ns002i007839"}' https://hvault.staging.my_company/v1/auth/kubernetes_demo_re_ns002i007839/loginFrom the pod I then get to make a successful secret retrieval with the follwing curlcurl -H "X-Vault-Request: true" -H "X-Vault-Namespace: DATA/DATAOPE/EC002I002621/" -H "X-Vault-Token: <token retrieved by previous command>" https://hvault.staging.my_company/v1/secret/data/testI try to make sense of the doc and cannot get to a working configurationI setup my HashiCorp Vault Client with the following properties direct properties

Hello,Has anyone ever used formal methods or a logic language such as TlaPlus to check Apache Ranger policies?If so, could you please provide examples?

How can I send select queries to the Oracle database to be binded with the executeSql processor in Apache NiFi 1.23.2 version?

Hey everyone, I’m currently preparing for the CCA175 (Cloudera Data Engineer) exam and focusing heavily on real, hands-on scenario challenges to strengthen my understanding. So far, I’ve practiced with various ingestion and transformation pipelines, but I’m stuck on one scenario that feels very close to what the actual exam might present. Midway through my study plan, I started using Certs Matrix, which has helped me evaluate different approaches to solving Spark and Hadoop workflow problems under pressure. The scenario I’m trying to clarify is this: If you receive streaming data in inconsistent formats and need to cleanse, transform, and store it efficiently in HDFS, which approach would be most exam-accurate using Spark Structured Streaming with schema evolution, designing separate ETL pipelines for each input format, or relying on a unified schema-on-read strategy? I’d really appreciate insights from anyone who has taken CCA175 or handled similar real-world pipelines. Your guidance

According to https://hive.apache.org/general/downloads/"The Apache Hive Community has voted to declare the 3.x release line as End of Life (EOL). This means no further updates or releases will be made for this series.We urge all Hive 3.x users to upgrade to the latest versions promptly to benefit from new features and ongoing support."CDP 7.3.1 is still very much based on Hive 3.https://docs.cloudera.com/cdp-private-cloud-base/7.3.1/hive-introduction/topics/hive_whats_new_in_this_release_hive.htmlhttps://docs.cloudera.com/cdp-private-cloud-base/7.3.1/hive-introduction/topics/hive-apache-hive-3-architectural-overview.htmlWhen will Cloudera on base support Hive 4?Thanks 

Hello Cloudera Community,I'm working on a proof-of-concept for migrating our client's existing Cloudera ( CDP 7.1.4 and Impala 3.4.0 ) environment to CDP 7.1.9 with Impala 4.0. We need to validate that our workloads running on the current version will be compatible with the newer version before proceeding with the production migration.I've downloaded the trial version, but it provides CDP 7.1.4 with Impala 3.4.0. For our migration testing, we specifically need CDP 7.1.9 with Impala 4.0 to ensure accurate compatibility testing.Could you please advise on:1. How to obtain trial access to CDP 7.1.9 with Impala 4.0?2. Whether the current trial license can be used with version 7.1.9?3. Access to the CDP 7.1.9 repositories for trial purposes?This is a time-sensitive POC that will directly influence our client's decision to proceed with the CDP upgrade.Thank you for your assistance.Best regards

We were looking at storing the provenance *.gz ( lucene ) files so we could have a record of all activity through Nifi, as our IT security people are fussy about such things.It seems Nifi writes to multiple files at the same time, so this may not be possible, or perhaps there is a better way of doing this? I would welcome thoughts as to how we might do this, or maybe propose a different way to achieve the same logging outcome.Thanks in advance.