Ask the Community

"Hi all, I am a Data Engineer looking to test and familiarize myself with the latest Cloudera Manager and Cloudera Runtime 7.x features. Since Cloudera Express is deprecated, does Cloudera offer any Developer License, Academic License, or a low-cost Non-Production subscription specifically for sandbox/R&D environments? Thank you!"

"Hi Community, our team is planning to evaluate CDP Private Cloud Base for our internal big data modernization project. We want to set up a small 3-node cluster for a Proof of Concept (PoC) on our local infrastructure. Since the repository is now password-protected, how can we officially request a temporary trial license key and repository credentials to test the installation with Cloudera Manager? Any guidance would be highly appreciated."

The CM version is 7.13.1. Airflow is not available as an option to "Add Service". Is it possible to use Cloudera's Airflow with 7.1.9sp1?If yes, where  can I download the CSD for Airflow?If there is no CSD, how do I install Cloudera's Airflow as a stand-alone service?Thank you.  

Apache NiFi 2.6.0/2.7.2 ListSFTP Hostname not working when using Proxy Configuration Service.   It works when an IP address is listed.Example:Hostname: cloudera.comPort: 22Proxy Configuration Service: StandardProxyConfigurationService (Proxy Type=HTTP)When executed (run once), it gets the following error:ListSFTP[id=a423ae96-019b-1000-c5c2-8e609a342bc5] Processing failed: org.apache.nifi.processors.standard.socket.ClientConnectException: SSH Connection failed [cloudera.com:22]- Caused by: org.apache.sshd.common.SshException: DefaultConnectFuture[A@cloudera.com/<unresolved>:22]: Failed (UnknownHostException) to execute: cloudera.com- Caused by: java.net.UnknownHostException: cloudera.comWhen using the IP address of cloudera.com as 151.101.127.10 will work (disregard the proxy 403 error as it is not white listed) ListSFTP[id=a423ae96-019b-1000-c5c2-8e609a342bc5] Processing failed: org.apache.nifi.processors.standard.socket.ClientAuthenticationException: SSH

I'm copying several directories with several files eachYarn application full termination seems affected by the number of object transferredIn this case we have 9 minutes between 100% completion and job termination26/06/05 10:47:20 INFO mapreduce.Job: map 100% reduce 0%26/06/05 10:56:25 INFO mapreduce.Job: Job job_1780210407885_1954 completed successfullywithFiles Copied=68744DIR_COPY=23058while here few seconds26/06/05 10:43:11 INFO mapreduce.Job: map 100% reduce 0%26/06/05 10:43:32 INFO mapreduce.Job: Job job_1780210407885_1958 completed successfullywithFiles Copied=12300DIR_COPY=191 Is the any explanation for such a behaviour?

Hi Team,I am exploring secret management options in Apache NiFi 2.8.0 and would like to understand whether NiFi supports retrieving passwords directly from HashiCorp Vault KV during startup.My use case is to store the following properties in Vault KV rather than in nifi.properties:- nifi.security.keystorePasswd- nifi.security.keyPasswd- nifi.security.truststorePasswdThe objective is to avoid storing these sensitive values in plain text on the NiFi host while still allowing NiFi to access them during the bootstrap/startup process.I am specifically interested in startup configuration properties that are required before the application is fully initialized.Has anyone successfully configured NiFi 2.8.0 to retrieve these passwords directly from HashiCorp Vault KV during startup? If so, are there any recommended approaches, examples, or best practices for this use case?Any guidance would be appreciated.Thank you.

Hello it seems templates which existed in NiFi 1 is removed in NiFi 2.8.0How to import the flow definition to the new instance? Earlier we were able to move nif flow to new instances of NiFi using templates.Any screenshot will be helpful for NiFi 2. Currently I can only download flow definition by right clicking a process group. Some article suggested Registry. Do we have any other approach like nifi 1 where we used to browse and upload.

Hello we have single user based NiFi but in reality around 15 to 20 people use that instance. How to know who added a process group or stopped a processor and at which point of time was that processor stopped? Do we have any IP getting recorded or anything like that? Is it possible in NiFi having OIDC/Oauth2 based authentication that which user added/removed something?

I just finished installing a new cluster but failed to start the zookeeper service. Each instance started but had the "bad health" flag. The zookeeper log on node 3 showed this error repeatedly. ++++Cannot open channel to 1 at election address node1/61.62.63.1:4181java.net.ConnectException: Connection refused++++On node 3, the connection to node 1 zookeeper port showed no issue.$ nc -zv node1 4181Ncat: Version 7.92 ( https://nmap.org/ncat )Ncat: Connected to 61.62.63.1:4181.Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds.On node1, the zookeeper port is open too$ ss -tulnp | grep 4181tcp LISTEN 0 50 61.62.63.1:4181 0.0.0.0:*The private cloud base cluster is 7.1.9 sp1.Thank you.Regards,

Hi everyone, I am Raghav Hinduja Swiss (Switzerland) based IT-professional. I’m trying to install Cloudera Manager on Ubuntu. If you have experience with this setup, could you please share your installation steps or any helpful tips?

Hello Cloudera Community,During the same Test cluster upgrade from Cloudera Runtime 7.1.9 to 7.3.2.0 GA (before Prod rollout), we encountered a persistent issue with Ranger UserSync, which we were not able to resolve using supported configuration or migration steps.EnvironmentOriginal Runtime: 7.1.9Upgraded Runtime: 7.3.2.0 (GA)Parcel: CDH-7.3.2-1.cdh7.3.2.p0.77083870Cloudera Manager: 7.11.3OS: RHEL / Oracle Linux 8.7Authentication: LDAP / KerberosUpgrade type: Test cluster before ProdAfter upgrade to 7.3.2.0, Ranger UserSync and RMS does not start, while Ranger Admin and Ranger Tagsync starts and works normally.UserSync fails immediately with error:Error: Could not find or load main classorg.apache.ranger.authentication.UnixAuthenticationServiceCaused by: java.lang.ClassNotFoundExceptionWe tried all supported and commonly recommended remediation steps:Deleting the existing Ranger UserSync role (originally created on 7.1.9);Re‑creating a new Ranger UserSync role;Configuring LDAP‑b

Hello Cloudera Community, We upgraded a Test cluster from Cloudera Runtime 7.1.9 to 7.3.2.0 GA (before Prod rollout) and encountered an issue where Apache Atlas fails to start after the upgrade.EnvironmentOriginal Runtime: 7.1.9Upgraded Runtime: 7.3.2.0 (GA)Parcel: CDH-7.3.2-1.cdh7.3.2.p0.77083870Cloudera Manager: 7.11.3OS: RHEL / Oracle Linux 8.7Cluster Java: OpenJDK 17 (required by 7.3.2)HDFS metadata upgrade: completedAfter upgrade, Atlas fails with:Unrecognized VM option 'CMSClassUnloadingEnabled' Error: Could not create the Java Virtual MachineThe generated JVM command line contains CMS GC flags: -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled.  These flags are not supported on Java ≥ 9.Atlas in 7.3.2.0 GA is compiled with Java 17, so running Atlas on Java 8 or 11 is also not possible: UnsupportedClassVersionError (class file version 61.0).Looks like CMS GC flags are hard‑coded in the parcel . We attempted multiple supported CM‑level overrides,

Hello can we secure our apis in nifi? Suppose we create nifi flow which utilises HandleHttpRequest for exposing an api. How to secure this? Is it possible to have Oauth2/Oidc?

I can get a Knox token from the Web UI in CDP Private Cloud, but is it possible to get it from the API without having to use the Web UI?

We have Nifi 2.5.0 and a problem whereby some files seem to be "stuck" after leaving a 3 node cluster, and get "stuck" in a site-to-site connection queue after the cluster.We ingest 60,000 small files out of a series of folders that are nested to a depth of 10 folders.Most files seem to move through the cluster OK, but we get a few that seem to enter the cluster fine , but then have a problem leaving it. Its different files each time and they sit there for 30-60 mins and eventually move on. Its not the same files each time.The network and compute resources are fine, the memory that Nifi is using is about 47% of the JVM maximum. Disk space doesn't seem to be an issue either. I had a look through the nifi-app.log file and nothing appears to show as an issue. Cluster heartbeat is average 10 milliseconds,  GC runs in 50 milliseconds every 30 seconds.  Would welcome some suggestions.Thank you in advance.

I'm exporting several tables and I observe that some files are copied to the target path using DistCp (slow) while other files with some other (fast) meanThere is no evidence of the rational behind the choice but the other odd thing is that even if a table is made up of multiple files, hive starts 1 DistCp for each file instead of passing the entire directoryIs there any option to drive the behaviour?

WARNING Port is being used by CM Agent: 7182ERROR Heartbeating to wblvhvbssuat16.telkom.co.za:7182 failed.

We have VM server on which we have NiFi installed within the container.When we try to connect using sftp command from within the docker it connects fine.docker exec -it nifi bashHowever when we try from NiFi PutSFTP processor, it fails with below errorRouting to failure since unable to transfer FlowFile[filename=TGT_TEST28042026.csv] to remote host sftp.AAAAA.com:net. schmizz. sshj. transport. TransportException: Connection reset- Caused by: java.net.SocketException: ConnectionHow and where do I get more details logs ?Also, is sftp command and NiFi PutSFTP works differently ?This is similar to my previous  posthttps://community.cloudera.com/t5/Support-Questions/SFTP-CLI-works-but-PutSFTP-fails/td-p/413903But this time vendor has whitelisted the IP and we are able to connect from within docker on which Nifi is running.ThanksGautam P

Hello I need help to setup oidc based login mechanism for NiFi. I came across this article https://docs.cloudera.com/cfm-operator/2.10.0/configure-nifi-cr/topics/cfm-op-configure-nifi-cr-oidc-auth.htmlbut I want to know how to do it for some provider other than Keycloak. I have an internal provider. Also when I read this article I saw some terms like Discovery url and "The clientID and clientSecret fields are provided to NiFi in a Kubernetes secret. Create that secret with the following command: kubectl create secret generic oidc-client-secret --from-literal=clientID=[***YOUR CLIENT ID***] --from-literal=clientSecret=[***YOUR CLIENT SECRET***]I am quite new to these terms and wanted to know how to do it in NiFi. Is kubernetes required? I have installed both NiFi 1.23 and NiFi 2.8 on a simple machine.

Facing the below issue on Impala - Failed to get minimum memory reservation of 429.94 MB on daemon hydwtbdn1.icicisecurities.corp:22000 for query b44fc02d59517c5b:40ea915f00000000 due to following error: Failed to increase reservation by 429.94 MB because it would exceed the applicable reservation limit for the "Process" ReservationTracker: reservation_limit=255.00 GB reservation=255.00 GB used_reservation=0 child_reservations=255.00 GB The top 5 queries that allocated memory under this tracker are: Query(2e4e69d2f55f0014:fe08e92f00000000): Reservation=153.90 GB ReservationLimit=240.00 GB OtherMemory=1.87 GB Total=155.77 GB Peak=155.85 GB Query(7b478b1891d4039d:8bb6141800000000): Reservation=101.08 GB ReservationLimit=240.00 GB OtherMemory=237.92 MB Total=101.31 GB Peak=101.32 GB Query(e5471eefc1b23d4c:36badde100000000): Reservation=0 ReservationLimit=240.00 GB OtherMemory=6.33 MB Total=6.33 MB Peak=925.69 MB Query(6046dfb9cc825b97:45bd0eef00000000): Reservation=0 ReservationLimit

Hello,We are trying to implement Impala Admission Control.Our requirement  is  if we have 5 users let it be A,B,C,D and E. I have created 2  resource pool and added user A,B,C under Submission Access Control of "resource_pool_1" and added user D and E under "resource_pool_2" now we require if user A or B or C will run any impala query it should run in "resource_pool_1" and if user D or E run any impala query it should run in "resource_pool_2". But user does not set pool while running query it should run in allocated pool directlyHow can we achieve this from  Impala Admission Control Configuration?Thanks in advance!

I'm running Nifi in ECS with an ALB. Everything is up and seems normal except there are intermittent 502 errors that pop up in the Nifi ui that look like this It seems that the source of the 502 is the load balancer but since it's intermittent it seems like an interaction between the load balancer and Nifi that is causing this behavior.I've tried messing with the following settings so that the idle timeout (60 seconds) on the ALB is lowernifi.web.request.timeoutnifi.remote.input.http.transaction.ttlI've also made sure the nifi.web.proxy.host configuration is set the hostname that I'm using to hit the loadbalancer. Has anyone else experienced something similar getting a load balancer in front of Nifi?

Previously we used NiFi 1.24, running it in OpenShift containers via the entrypoint "./bin/nifi.sh stateless...". Now I'm trying to migrate the dataflow to version 2.8.0 and am encountering an error - the "stateless" parameter for nifi.sh is now missing. I read all the documentation in the NIFI release notes and did not find information about this anywhere. Git still says that you can run it this way (nifi-stateless-assembly readme).Tell me, is there any possibility to run the NiFi 2.8.0 distribution via the console in stateless mode? How to use a separate nifi-stateless distribution?

Hello I need to invoke a url which is protected by oauth2. I know to invoke simple apis but not with oauth2. How can we make a flow?

Hi everyone,I’m facing an issue while initializing a compute cluster on an existing CDP Public Cloud environment (AWS) using restricted IAM policies.The operation fails with the following error:IAM Restricted Resource Policy validation cannot be completed on AWS:secret encryption is enabled but the secret encryption KMS key is not provided.Liftie does not have the permission to create the KMS key.Please provide a valid Customer Managed Key for secret encryption In the other experience (CDE,CDF) I bypassed this error by adding skip validation. Unfortunately, skip validation is not possible when activating the compute cluster. I've modified also the cross account role by adding the action as mentioned on paragraph "For Let CDP generate CMK" https://docs.cloudera.com/dataflow/cloud/aws-requirements/cdf-aws-requirements.pdf but nothing to do. Do you have any suggestion? Thanks