Solved

Hi everyone. I created 2 queue yarns in cdp 7.1.8. Precisely I created 2 queues: users queue --> configured capacity 50% and maximum capacity 100% hive queue -->configured capacity 50% and maximum capacity 100% I enabled child queues mode for all 2 queues with the following configuration parameters:Dynamic Queue Minimum User Limit100%Dynamic Queue User Limit Factor1Dynamic Queue Maximum Applications1000Dynamic Queue Maximum AM Resource Limit20%Dynamic Queue Ordering PolicyFairThe problem we encounter is that if even just 2 queries are executed in the same queue the second goes pending and is not executed in parallel, I show an example belowAnother example would be an oozie job that only calls a spark job. The oozie job runs infinitely waiting for the pyspark to run. However, in the default queue this problem is not present.The development cluster is made up of 3 worker nodes (144 vcores 180 gb yarn memory).Can you provide us support or optimizations to do at the queue configurati

Hello Team,I am unable to run the import command for Hive, since it is failing with error invalid path.I copied the table content to another folder and it has permission for hive user, but while running import command it says invalid path. i ran import command on the path where the source table location still it fails with same reason.Beeline version 3.1.3000.7.1.8.28-1 by Apache Hive[root@machine1 Log_Files]# sudo -u hive hadoop fs -ls /warehouse/tablespace/managed/hive/demo.db/test1/*Found 1 items-rw-r--r-- 3 hive hive 621 2024-05-02 16:05 /warehouse/tablespace/managed/hive/demo.db/test1/delta_0000004_0000004_0000/bucket_00000_0[root@machine1 Log_Files]# sudo -u hive hadoop fs -ls /jay123/restore_jobid123243/jayesh/*Found 1 itemsdrwxr-xr-x - hive hive 0 2024-05-02 16:05 /jay123/restore_jobid123243/jayesh/test1/delta_0000004_0000004_0000[root@machine1 Log_Files]# sudo -E -u hive beeline -u "jdbc:hive2://machine1.testride.com:2181,machine22.testride.com:2181/;serviceDiscoveryMode=zooKe

I am using Apache nifi invokeHttp processor to call Google API. The url contains a path which i need to encode.So in the URL,only path is encoded e.g. resources%2F8cb90ae6-16cc-4460-99f4-3ae0689b3b66%2Fdata_asset%2FCETIvowels-38.pdfHowever whenver I set encoded url, the processor internally replaces %2F with %252F which results in HTTP 404 response.E.g. Nifi convert URL - https://storage.googleapis.com/storage/v1/b/t-bis-1hg-jk/o/resources%2F8cb90ae6-16cc-4460-99f4-3ae0689b3b66%2Fdata_asset%2FCETIvowels-38.pdf?alt=mediatohttps://storage.googleapis.com/storage/v1/b/t-bis-1hg-jk/o/resources%252F8cb90ae6-16cc-4460-99f4-3ae0689b3b66%252Fdata_asset%252FCETIvowels-38.pdf?alt=media 

I am getting the below error while installing the agent. I am working on Ubuntu 20.04. It's a single machine cluster I want to make. Can anyone help me out with this? Cleaning the package manager cache... View Failed to acquire installation lock. View/tmp/scm_prepare_node.OHxqRMPmusing SSH_CLIENT to get the SCM hostname: 10.4.47.28 49784 22opening logging file descriptorStarting installation script...Acquiring installation lock.../tmp/scm_prepare_node.OHxqRMPm/scm_prepare_node.sh: line 1415: /tmp/.scm_prepare_node.lock: Permission deniedcould not open lock file, giving upwaiting for rollback request detected rollback requestrolling back installationReverting changes...rollback startedRemoving package repositories...Cleaning the package manager cache...unknown packager , exitingclosing logging file descriptor  

How to enable IAM for airflow either as ldap or oauth2. Please suggest

Hi team,I update airflow to v2.6.3, and find I can't connect to mssql 2008 by using JDBC.Following are my setting：Docker Desktop：v4.19.0airflow：v2.6.3JAVA：openjdk-11-jdkJDBC Driver：jtds-1.3.1.jarConnection URL：jdbc:jtds:sqlserver://xxx.xxx.xx.xx:1433/xxxxDriver Path：/usr/local/airflow/jars/jtds-1.3.1.jarDriver Class：net.sourceforge.jtds.jdbc.Driver The connection was failed.  I wonder why the message show the connection string as below：   jdbc://jdbc:jtds:sqlserver:  why not only  jdbc:jtds:sqlserver:  Any help will be appreciated, thanks!   

Hello all,We are having an issue while adding airflow service to the CDH 5 cluster.Problem: Executing the command "Upgrade Airflow DB" on the Airflow service in a Cloudera cluster on GCP (CentOS 7, CDH 5.16.2, Airflow 1.10.1) results in an error: "Could not create process: null". The attempt to upgrade the Airflow database fails while adding airflow service to the CDH cluster.Context:The same infrastructure and parcel configurations have been successfully used on 6-7 on-premises clusters.Logs are empty, and the scm-server log provides the same error message, indicating a failure to create the process.Environment:Operating System: CentOS 7CDH Version: 5.16.2Airflow Version: 1.10.1Steps Taken:Added the Airflow parcels and csd in the proper location with cloudera-scm owner.Added the Airflow service.Created Mysql DB for airflow.Added Mysql lib/client on the all hosts.Added Redis server to manage the airflow in cloudera.Checked logs for detailed error information, but logs are empty.Verifie

I came across this:Starting with the NiFi 1.0 release, a Zero-Leader Clustering paradigm is employed. Each node in a NiFi cluster performs the same tasks on the data, but each operates on a different set of data. Apache ZooKeeper elects a single node as the Cluster Coordinator, and failover is handled automatically by ZooKeeper. All cluster nodes report heartbeat and status information to the Cluster Coordinator. The Cluster Coordinator is responsible for disconnecting and connecting nodes. Additionally, every cluster has one Primary Node, also elected by ZooKeeper.If nifi follows zero-leader paradigm, then what is the purpose of primary node? And how does it differ from other nodes?

Can we run Cloudera Data Platform (CDP) on Amazon Elastic Container Service (ECS) within an AWS environment?

Hi,I require some assistance with CDP -PC CDE cli command. What I am trying to achieve is to run cde jobs via cde cli with the current days date.My command line is - cde job run --name generate_post_view --vcluster-endpoint https://my.cluster.com/dex/api/v1 --user my.user_id Would need assistance on how to pass date argument in this command line so that I don’t have to go to CDE UI and edit the date for running every job.Appreciate any help in thisThanks 

I've created a cluster in Cloudera and now I want to add Hive, but I need it to utilize the Tez execution engine. So, I'm adding Hive on Tez and the Tez service. However, I'm confused about assigning server IPs to roles.In the Hive service section, I see options for gateway, hiveserver2, metastore, and WebHcat.In the Hive on Tez section, I see options for gateway and hiveserver2.In the Tez section, only gateway is listed.Now, I'm confused. Should I assign the same IP to all gateways and hiveserver2, or should I assign different IPs? Where should I assign hiveserver2, in Hive or in Hive on Tez? And how many servers can I assign to hiverserver2, metastore, and gateway?Moreover, when the Hive section covers gateway and hiverserver2 options, why do the Hive on Tez and Tez sections also show these options?Additionally, I would appreciate clarification on how to configure Hive high availability in this setup. I'm seeking guidance to proceed and would appreciate any help from the Cloudera com

Hi,I'm not sure if this has been asked before but Im finding it strange that there is not much info or discussion about it. Basically I have scenario where at some point of time I believe I was getting some corrupted data from an API call. When I went to verify that by executing the API call after few hours from the error I dont see the corrupted data. How do I prove\disapprove  this? if I can search the data provenance for that particular response flowfile I would be able to see what did i get at that time after the call. The problem is the out of the box search provenance criteria doesn't provide a way to search against the content or the flowfile custom attributes and it only allows to search against system fields attributes that I dont store. Is there a way to perform such search even by creating some dataflow using certain processors in nifi or using some scripting language?@MattWhoor any body who can help with I would really appreciated. 

 Failed Provisioning while provisioning CML work space. It was succeeded when the first time I tried.However failed provisioning, after re-porovisioning after delete workspace on my test-bed.And also It happened on customer site for POC. I tried version 1.5.2 and 1.5.3, 1.5.3-h1 on RHEL 8.4.Version of Cloudera Manager Cloudera Manager 7.11.3 (#50275000 built by jenkins on 20240213-1404 git: 14e82e253ab970bfd576e4f80d297769a527df18)Tried DataServices Version1.5.2-b886 / 1.5.3-b279 / 1.5.3-h1-b2 Version of screen shot 1.5.2-b886 I have to update the Data services on customer site.Please give any advices.

Hello,I tried to install a trial CDP Cluster following the given steps: https://docs.cloudera.com/cdp-private-cloud-base/latest/installation/topics/cdpdc-trial-installation...The Cloudera Manager executed is https://archive.cloudera.com/cm7/7.4.4/cloudera-manager-installer.bin It was installed successfully.The Cluster installation is failing with the messages:Uninstalled. View Failed to install openjdk8 package. Error: Cannot find a valid baseurl for repo: cloudera-manager This is the complete log:Uninstalled. View Failed to install openjdk8 package. View/tmp/scm_prepare_node.pCEw7JVQusing SSH_CLIENT to get the SCM hostname: 100.200.196.179 59194 22opening logging file descriptorStarting installation script...Acquiring installation lock...BEGIN flock 4END (0)Detecting root privileges...effective UID is 1000BEGIN which pbrunEND (1)BEGIN sudo -S id/usr/bin/which: no pbrun in (/home/opc/.local/bin:/home/opc/bin:/usr/share/Modules/bin:/usr/local/bin:/usr/bin:/usr/l

Greetings, everyone.I'm kind of new to NiFi and trying to implement some simple authorization method that is not default (single-user). All I want to do is a couple static users with some permissions and login screen asking for credentials - 2 roots and 2 read-only will be more than enough for example.I've tried to search for my answer in other topics both in cloudera and documentation, but didn't find any useful info. As far as I'm concerned, Nifi currently support 3 methods - single-user, LDAP and Kerberos.Do I really need to use some LDAP to make this happen or there's more simplified solution in that case? If so - can anyone share some ideas what exactly I need to do?I use Nifi 2.0.0-M2 at the moment. Let me know if some additional info is needed.Thanks a lot in advance and have a great day!

HI,Does Apache storm still have support in cloudera CDP?if it is supported,  please mention methods to install it in CDP Public cloud.I tried installing form external source but facing kerberos issue on connecting HDFS Bolt with secure HDFSplease help on this 

I have installed Livy service in Cloudera CDP (7.1.7) cluster, accessing the admin web ui throwing 403 error. See the details below:   

My data flow starts from a single FlowFile produced by a Sqoop job, which then expands into multiple FlowFiles after executing the GetHDFSFileInfo Processor (based on the number of HDFS files). To capture all failure scenarios, I have created a Child processor group with the following processorsInput Port --> GetHDFSFileInfo --> RouteOnAttribute --> UpdateAttribute --> FetchHDFS --> PutSFTP --> ModifyBytes --> Output PortMain Processor Group--------------------RouteOnAttribute --> Above mentioned Child Processor Group --> MergeContent --> downstream processorsThe child processor group is configured with "FlowFile Concurrency = Single FlowFile Per Node" and "Outbound Policy = Batch Output" to ensure that all fetched FlowFiles are successfully processed (written to the SFTP server).My GetHDFSFileInfo processor returns 20 HDFS files, and each execution successfully transfers 18 to 19 files to my SFTP server. However, during each execution, one or two file tra

I faced failed Initialize embedded Vault shile installing DataServices.It happened everytime evenif I try install again from start if it happened once. I tried install on system below- Red Hat Enterprise Linux release 8.4 (Ootpa)- Cloudera Manager 7.11.3 (#50275000 built by jenkins on 20240213-1404 git: 14e82e253ab970bfd576e4f80d297769a527df18)- 1.5.2-b886-ecs-1.5.2-b886.p0.46792599 / 1.5.3-b297-ecs-1.5.3-b297.p0.50802651 both I tried stdoutFri Apr 12 11:36:52 KST 2024Running on: cdppvc1.hostname.com (192.168.10.10)JAVA_HOME=/usr/lib/jvm/java-openjdkusing /usr/lib/jvm/java-openjdk as JAVA_HOMEnamespace/vault-system createdhelmchart.helm.cattle.io/vault createdcertificatesigningrequest.certificates.k8s.io/vault-csr createdcertificatesigningrequest.certificates.k8s.io/vault-csr approvedsecret/vault-server-tls createdsecret/ingress-cert createdhelmchart.helm.cattle.io/vault unchangedWait 30 seconds for startup...Timed out waiting for vault to come up stderr++ kube

Hello Experts,I am using Apache Nifi 1.25 now and have a requirement to consume data from a specific offset of a topic partition.I see in Nifi ConsumeKafka processor to configure only "earliest" or "latest" but not specific offset number, is there any way to achieve this ? Any info/suggestion would be appreciated. Thanks,Mahendra

Hi everyone, I would like some clarification. Thank you.CDP recommends disabling the 'tuned' service for performance optimization.Disable the tuned ServiceThe suggestion to disable 'tuned' stems from its capability to transition the CPU into c-state idle mode, potentially leading to decreased performance.Disable the tuned Service on RHEL 8 We would like some clarification about the explanation regarding 'tuned', such as its potential impact on Hadoop. Thank you. For example, THP:Disabling Transparent Hugepages (THP) Most Linux platforms supported by Cloudera Runtime include a feature called transparent hugepages, which interacts poorly with Hadoop workloads and can seriously degrade performance.For example, swappiness:Setting the vm.swappiness Linux Kernel ParameterOn most systems, vm.swappiness is set to 60 by default. This is not suitable for Hadoop clusters because processes are sometimes swapped even when enough memory is available. Regards

 What is correct?

I am new to this platform, and I need your help to get a complete case study (that includes spark MLlib and other tools to classify the data for example in Twitter) to apply in my environment.

Hi.First of all, I'm not good at English.So please understand my English 🙂  Now, I have no any application on YARN.But, "Used Application Master Resources" isn't empty!!! Especially, when actual applicaion is running, a lot of job become pending state... Is there great guy to help me? ㅠ_ㅠ  Thank you to read my question.

hello cloudera community, we have a problem we are not able to view the execution logs of the jobs that were executed in yarn by the resourcemanager of yarn when we click on logs it presents the error: "Error getting logs at hostname:8041" how can we solve this problem? we are using cdh express 5.16.2