Q&A

I have copied hive table from Prod to QA by using get and put command but while executing select counnt (*) is it showing 0 count but i can see physical location is having data however i have checked all permission are in place so i did invalidate metadata and table refresh and msck repiare but still facing same issue. can seomeone help. RegardsSaim

Hello TeamWe have an increasing number of applications being added to our NiFi server. Now the nifi has got too slow that it won't load the home page or when we click a process group then it won't open despite browser url showing the process group id as if page should have gone there.Also the apis we have implemented using HandleHttpRequest return Socket Timeout error.Please note these were working fine earlier. What should we do?

I have set my impala.admission-control.min-query-mem-limit.root.default to 2GiB and my impala.admission-control.max-query-mem-limit.root.default to 57GiB, which aligned with the memory allocation for each Impala Executor node that I've set on the resource templates. My problem is, when I'm running a heavy query (I tried to set it so the limit is 10000000), the limit memory is set to 2GiB and makes the query OOM, which is strange because the max-query-mem-limit is set to 57GiB, so the limit should be > 2GiB and <= 57GiB if the query is heavy, right? Yet it's just set to the minimum mem limit. I know that the limit is set to clamp(min memory limit per query, memory estimated, max memory limit per query), but the estimated memory is 4GB and what makes it stranger is that when I try to define SET MEM_LIMIT in the query to another number like 4G or 8G, the estimated memory changes; which means there's some calculation for the estimated memory based on the memory li

I missed my CDP-0011 certification appointment. What should I do?The certification fee is very expensive for me, so I cannot repurchase it. Has this situation happened to any of you, and what is the correct action I should take?

 Hi all,I am working with Cloudera Data Visualization 7.1.6 and creating a dashboard that includes a KPI visual.The visual uses a dataset based on a Hive query. Is there any way to pass dashboard filter values or parameters directly to the dataset query?Based on the official documentation and the information I found online, it appears that Cloudera Data Visualization applies filters only at the visualization or UI level and does not support passing parameters to the underlying dataset query.Could someone confirm whether this is an architectural limitation, or whether there is an alternative solution or workaround?Thanks in advance for any suggestions or help.

HiWe are having problems with a single instance of nifi 2.5.0 on a physical windows 11 machine with 64 CPU cores, 64 GB RAM and NVME 12 TB ( 4 x 3 TB drives ) striped D : drive set ingesting slowly.  C: drive is for windows.I was hoping someone could maybe give a sequential list of things to check so we at least are approaching this the right way.The people who set up the machine have put the ingest directory ( ingest files via Getfiles ), and the nifi instance ( content repo and everything else ) on the same single D ; drive striped disk set.  Being a striped set I wouldn't have thought speed would be an issue even with both ingest and content repo on the same striped set. So one thing I wondered was can we force nifi to use all cores, or do we need keep some cores  ( maybe 4 ? ) for windows to use? And if we did use say 60 cores, what is the best way to do this -  and is it set at a nifi processor level,  or do you need to force Nifi to use all core

I think I might just be misunderstanding something, because it seems like NIFI requires HTTPS for intra cluster communication and uses the SAME HTTPS channel for external UI and API access?I'm expecting to have intra cluster communication be separate from my load balancer/ingress. Meaning the nodes communicate securely with each other but I can set the API and UI to HTTP so I can terminate that at my load balancer- this is very standard practice.What I think I'm seeing is NIFI uses the same HTTPS channel for both internal and external communication? That can't be right because there is no way I'm registering all my NIFI nodes with a public DNS domain.But this can't be the case right? I should be able to set just the end user API and UI to HTTP and use a self signed cert for intra-node communication.I can't even seem to be able to just disable HTTPS entirely, nifi's config is poorly documented and disabling HTTPS seems to break everything. I have this on a secured cluster and because th

 I have a question regarding the long-term support alignment for CDP Private Cloud Base 7.1.9.As you know, CDP 7.1.9 is an LTS release supported until October 2028. Initially, this version only supported Python 3.8, but due to the relatively early EOL of Python 3.8, support for Python 3.11 was introduced starting from SP1 CHF9.On the other hand, CDP 7.1.9 depends on OpenJDK 17, whose end of support is currently December 31, 2027. This creates a potential misalignment with the CDP LTS support period extending into late 2028.Given the precedent with Python (i.e., adding support for a newer version within the same LTS release via CHF updates), we would like to understand the roadmap for Java:Are there any plans to support a newer OpenJDK version (e.g., OpenJDK 21 or later) within CDP 7.1.9 through future SP/CHF updates?If so, is there an estimated timeline or target CHF/SP where this support might be introduced?If not, how does Cloudera plan to address the gap between the OpenJDK 17

We are running a sparkly job that runs queries on cloudera CDP hive cluster. The job sometimes stops before a dbwriteTable function, doing nothing and running indefinitely. The job doesn't always stop in the same point, but always during this trywrite function invocation, not catching any error:trywrite = function(sc, new_name, df, log_obj, wait_sec = 600, max_wait = 3600) { start_time = Sys.time() while (difftime(Sys.time(), start_time, units = 'secs') <= max_wait) { print(paste0('Attempt to write table: ', new_name, ' - ', Sys.time())) # Connection is valid? if (!DBI::dbIsValid(sc)) { error(log_obj, paste0('Connection not valid during write table: ', new_name)) stop(paste0('Failed to write table: ', new_name)) } tryCatch({ print(paste0('Writing table: ', new_name)) result = DBI::dbWriteTable(sc, new_name, df) print(paste0('Write completed table: ', new_name, ' - ', Sys.time())) return(result) }, error = function(

During my preparation, I used multiple learning resources, including official Cloudera documentation, hands-on practice labs, and practice questions, which helped me assess my knowledge and identify areas that needed improvement. Some tips for candidates preparing for the exam: Focus on understanding concepts rather than memorization.Gain hands-on experience with Cloudera technologies.Review the official exam objectives carefully.Take practice tests to evaluate your readiness.Spend extra time on troubleshooting and real-world scenarios. I would like to thank the Cloudera community for the valuable discussions and learning resources that contributed to my success. Best wishes to everyone currently preparing for their Cloudera certification exams. Keep learning and stay focused—you can do it!

 **Environment**- Cloudera CFM 2.1.7 / Apache NiFi 1.26.0- Cloudera CDP 7.3.1- Flow source: Cisco ASA Firewall (IPFIX export) **Issue** The `ListenNetFlow` processor successfully decodes standard IPFIX/NetFlow v9 fields but produces a WARN for every packet that contains Cisco ASA enterprise fields: ```WARN com.cloudera.nifi.netflow.ListenNetFlow: Sender [/x.x.x.x] Field Type [33002] Length [2] Field Decoder not foundWARN com.cloudera.nifi.netflow.ListenNetFlow: Sender [/x.x.x.x] Field Type [33001] Length [12] Field Decoder not foundWARN com.cloudera.nifi.netflow.ListenNetFlow: Sender [/x.x.x.x] Field Type [33000] Length [12] Field Decoder not foundWARN com.cloudera.nifi.netflow.ListenNetFlow: Sender [/x.x.x.x] Field Type [40000] Length [12] Field Decoder not found ``` At production NetFlow volume, this generates thousands of WARN lines per second, making `nifi-app.log` effectively unusable for monitoring other issues. More critically, the data in thes

"Hello everyone, I am researching the licensing model for Cloudera Private Cloud. Could someone clarify the baseline metrics for an On-Premises deployment? Is the subscription billed per Node or per CPU Core? Also, is there a minimum number of nodes required for a starting enterprise subscription? We want to budget this for our upcoming infrastructure planning."

"Hi all, I am a Data Engineer looking to test and familiarize myself with the latest Cloudera Manager and Cloudera Runtime 7.x features. Since Cloudera Express is deprecated, does Cloudera offer any Developer License, Academic License, or a low-cost Non-Production subscription specifically for sandbox/R&D environments? Thank you!"

"Hi Community, our team is planning to evaluate CDP Private Cloud Base for our internal big data modernization project. We want to set up a small 3-node cluster for a Proof of Concept (PoC) on our local infrastructure. Since the repository is now password-protected, how can we officially request a temporary trial license key and repository credentials to test the installation with Cloudera Manager? Any guidance would be highly appreciated."

The CM version is 7.13.1. Airflow is not available as an option to "Add Service". Is it possible to use Cloudera's Airflow with 7.1.9sp1?If yes, where  can I download the CSD for Airflow?If there is no CSD, how do I install Cloudera's Airflow as a stand-alone service?Thank you.  

Apache NiFi 2.6.0/2.7.2 ListSFTP Hostname not working when using Proxy Configuration Service.   It works when an IP address is listed.Example:Hostname: cloudera.comPort: 22Proxy Configuration Service: StandardProxyConfigurationService (Proxy Type=HTTP)When executed (run once), it gets the following error:ListSFTP[id=a423ae96-019b-1000-c5c2-8e609a342bc5] Processing failed: org.apache.nifi.processors.standard.socket.ClientConnectException: SSH Connection failed [cloudera.com:22]- Caused by: org.apache.sshd.common.SshException: DefaultConnectFuture[A@cloudera.com/<unresolved>:22]: Failed (UnknownHostException) to execute: cloudera.com- Caused by: java.net.UnknownHostException: cloudera.comWhen using the IP address of cloudera.com as 151.101.127.10 will work (disregard the proxy 403 error as it is not white listed) ListSFTP[id=a423ae96-019b-1000-c5c2-8e609a342bc5] Processing failed: org.apache.nifi.processors.standard.socket.ClientAuthenticationException: SSH

I'm copying several directories with several files eachYarn application full termination seems affected by the number of object transferredIn this case we have 9 minutes between 100% completion and job termination26/06/05 10:47:20 INFO mapreduce.Job: map 100% reduce 0%26/06/05 10:56:25 INFO mapreduce.Job: Job job_1780210407885_1954 completed successfullywithFiles Copied=68744DIR_COPY=23058while here few seconds26/06/05 10:43:11 INFO mapreduce.Job: map 100% reduce 0%26/06/05 10:43:32 INFO mapreduce.Job: Job job_1780210407885_1958 completed successfullywithFiles Copied=12300DIR_COPY=191 Is the any explanation for such a behaviour?

Hi Team,I am exploring secret management options in Apache NiFi 2.8.0 and would like to understand whether NiFi supports retrieving passwords directly from HashiCorp Vault KV during startup.My use case is to store the following properties in Vault KV rather than in nifi.properties:- nifi.security.keystorePasswd- nifi.security.keyPasswd- nifi.security.truststorePasswdThe objective is to avoid storing these sensitive values in plain text on the NiFi host while still allowing NiFi to access them during the bootstrap/startup process.I am specifically interested in startup configuration properties that are required before the application is fully initialized.Has anyone successfully configured NiFi 2.8.0 to retrieve these passwords directly from HashiCorp Vault KV during startup? If so, are there any recommended approaches, examples, or best practices for this use case?Any guidance would be appreciated.Thank you.

Hello it seems templates which existed in NiFi 1 is removed in NiFi 2.8.0How to import the flow definition to the new instance? Earlier we were able to move nif flow to new instances of NiFi using templates.Any screenshot will be helpful for NiFi 2. Currently I can only download flow definition by right clicking a process group. Some article suggested Registry. Do we have any other approach like nifi 1 where we used to browse and upload.

Hello we have single user based NiFi but in reality around 15 to 20 people use that instance. How to know who added a process group or stopped a processor and at which point of time was that processor stopped? Do we have any IP getting recorded or anything like that? Is it possible in NiFi having OIDC/Oauth2 based authentication that which user added/removed something?

I just finished installing a new cluster but failed to start the zookeeper service. Each instance started but had the "bad health" flag. The zookeeper log on node 3 showed this error repeatedly. ++++Cannot open channel to 1 at election address node1/61.62.63.1:4181java.net.ConnectException: Connection refused++++On node 3, the connection to node 1 zookeeper port showed no issue.$ nc -zv node1 4181Ncat: Version 7.92 ( https://nmap.org/ncat )Ncat: Connected to 61.62.63.1:4181.Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.On node1, the zookeeper port is open too$ ss -tulnp | grep 4181tcp LISTEN 0 50 61.62.63.1:4181 0.0.0.0:*The private cloud base cluster is 7.1.9 sp1.Thank you.Regards,

Hi everyone, I am Raghav Hinduja Swiss (Switzerland) based IT-professional. I’m trying to install Cloudera Manager on Ubuntu. If you have experience with this setup, could you please share your installation steps or any helpful tips?

Hello Cloudera Community,During the same Test cluster upgrade from Cloudera Runtime 7.1.9 to 7.3.2.0 GA (before Prod rollout), we encountered a persistent issue with Ranger UserSync, which we were not able to resolve using supported configuration or migration steps.EnvironmentOriginal Runtime: 7.1.9Upgraded Runtime: 7.3.2.0 (GA)Parcel: CDH-7.3.2-1.cdh7.3.2.p0.77083870Cloudera Manager: 7.11.3OS: RHEL / Oracle Linux 8.7Authentication: LDAP / KerberosUpgrade type: Test cluster before ProdAfter upgrade to 7.3.2.0, Ranger UserSync and RMS does not start, while Ranger Admin and Ranger Tagsync starts and works normally.UserSync fails immediately with error:Error: Could not find or load main classorg.apache.ranger.authentication.UnixAuthenticationServiceCaused by: java.lang.ClassNotFoundExceptionWe tried all supported and commonly recommended remediation steps:Deleting the existing Ranger UserSync role (originally created on 7.1.9);Re‑creating a new Ranger UserSync role;Configuring LDAP‑b

Hello Cloudera Community, We upgraded a Test cluster from Cloudera Runtime 7.1.9 to 7.3.2.0 GA (before Prod rollout) and encountered an issue where Apache Atlas fails to start after the upgrade.EnvironmentOriginal Runtime: 7.1.9Upgraded Runtime: 7.3.2.0 (GA)Parcel: CDH-7.3.2-1.cdh7.3.2.p0.77083870Cloudera Manager: 7.11.3OS: RHEL / Oracle Linux 8.7Cluster Java: OpenJDK 17 (required by 7.3.2)HDFS metadata upgrade: completedAfter upgrade, Atlas fails with:Unrecognized VM option 'CMSClassUnloadingEnabled' Error: Could not create the Java Virtual MachineThe generated JVM command line contains CMS GC flags: -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled.  These flags are not supported on Java ≥ 9.Atlas in 7.3.2.0 GA is compiled with Java 17, so running Atlas on Java 8 or 11 is also not possible: UnsupportedClassVersionError (class file version 61.0).Looks like CMS GC flags are hard‑coded in the parcel . We attempted multiple supported CM‑level overrides,

Hello can we secure our apis in nifi? Suppose we create nifi flow which utilises HandleHttpRequest for exposing an api. How to secure this? Is it possible to have Oauth2/Oidc?