Solved

I have two folders in hdfs with same name under the same directory. I want to get rid of one of them, but how do i do that?

I will use Spark2 in CDP and need to install Python3. Do I need to installation Python3 on every node in the CDP cluster, just only need to install it on one particular node? Spark2 job is executed in JVM containers that could be created on any worker node. I wonder whether the container is created upon a template? If yes, then how the template is created and where is it? Thanks. 

Hello everyone I have development cluster where I lost my ambari database which was on  a mysql database. Is there anyway to reinstall ambari server and have my cluster back up again ?   Thank you for your help

In the cdh6/cdh7 parcels I do not see the lib/hbase/bin/start-hbase.sh script.In cdh5 we used to use hbase-1.2.0-cdh5.16.1/bin/start-hbase.sh to start hbase in pseudo distributed mode.Can someone let me know what is the alternative to using hbase in pseudo distributed mode in CDH6/CDH7 ? Thanks,Sudheendra S.

We have a need to serve our Namenode UI through a proxy and prevent direct access to the UI.   There seems to be 2 ways we could do that but I'm having difficulty in implementing.1.  Set dfs.namenode.http-address to 127.0.0.1 :9870   Unfortunately, it appears that cloudera's implementation does not allow an address in dfs.namenode.http-address, but instead only accept a port number.  This is then combined with the namenode hostname to build the http address.   I attempted to override via a safety valve configuration but that appears to be intercepted as well.2.   Configure the jetty-web.xml with an IPAccessHandler that only allows connections from the local server.   I'm having difficulty finding where in the filesystem I'd set this so that cdh6 will pick this up for the namenode role.This is  Cloudera Enterprise 6.3.3 on Centos 7

When I rerun workflow with ooizeEnvironment detial：CDH6.2.1  

Nothing happens when I push download and go there:https://www.cloudera.com/downloads/hortonworks-sandbox/hdp.htmlSome time ago it worked, now I can't see any way to download the sandbox.

CDH 6.3.2 PHOENIX-5.0.0-cdh6.2.0.p0.1308267 add service phoenix   <property><name>hbase.regionserver.wal.codec</name><value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value></property> <property> <name>phoenix.functions.allowUserDefinedFunctions</name> <value>true</value> <description>enable UDF functions</description></property> adv service and adv client set phoenix.schema.mapSystemTablesToNamespace=true set phoenix.schema.isNamespaceMappingEnabled=true   when I run phoenix-sqlline   hbase region is OutOfMemory   2021-09-09T11:43:01.362+0800: [GC (CMS Initial Mark) [1 CMS-initial-mark: 34175K(34176K)] 48788K(49536K), 0.0309811 secs] [Times: user=0.04 sys=0.01, real=0.03 secs] 2021-09-09T11:43:01.393+0800: [CMS-concurrent-mark-start]2021-09-09T11:43:01.398+0800: [Full GC (Allocation Failure) 2021-09-09T11:43:01.398+0800: [CMS2021-09-09T11:43:01.414+0800

Hi experts,I ran a hive query using tez via beeline to Join tables and got the below error. 2021-09-08T17:07:55,932 INFO [HiveServer2-Background-Pool: Thread-140] hooks.ATSHook: Created ATS Hook2021-09-08T17:07:55,933 INFO [HiveServer2-Background-Pool: Thread-140] ql.Driver: Query ID = hive_20210908170755_9492c1e6-50ee-48da-8353-e49138d8b5272021-09-08T17:07:55,933 INFO [HiveServer2-Background-Pool: Thread-140] ql.Driver: Total jobs = 12021-09-08T17:07:55,933 INFO [HiveServer2-Background-Pool: Thread-140] ql.Driver: Launching Job 1 out of 12021-09-08T17:07:55,933 INFO [HiveServer2-Background-Pool: Thread-140] ql.Driver: Starting task [Stage-1:MAPRED] in serial mode2021-09-08T17:07:55,933 INFO [HiveServer2-Background-Pool: Thread-140] tez.TezSessionPoolManager: QueueName: null nonDefaultUser: false defaultQueuePool: null hasInitialSessions: false2021-09-08T17:07:55,933 INFO [HiveServer2-Background-Pool: Thread-140] tez.TezSessionPoolManager: Created new tez session for queue: null w

Hi,  we have a large number of flowfiles received and processed via NiFi.  Post the NiFi processing we then perform/send the data on via http request. The issue we have at the moment, is that the application (externally to NiFi) can not consume the data fast enough when received from NiFi. What I am looking for is, an ability to only take 1 flowfile every 5 seconds and everything in-between delete/remove.     Have looked at 1 processor that allows for a per flowfile process, however data then starts getting stuck in the queue.  So although processing 1 file every 5 seconds works, its the delete/remove of any other flow files that I need.   Thanks.

We are still running an old version of Clouder, v 5.15.0 (almost at the end of work upgrading) and a few hours ago users reported queries in Hue were not running. CM reported a number of issues so we decided to restart the entire cluster.   Zookeeper will not start, the error seen is below. There is a timeout issue that we've not been able to solve. There were no changes made at the time.    7:49:47.620 AM INFO QuorumPeerConfigReading configuration from: /run/cloudera-scm-agent/process/20394-zookeeper-server/zoo.cfg7:49:47.631 AM INFO QuorumPeerConfigDefaulting to majority quorums7:49:47.634 AM INFO DatadirCleanupManagerautopurge.snapRetainCount set to 37:49:47.634 AM INFO DatadirCleanupManagerautopurge.purgeInterval set to 247:49:47.635 AM INFO DatadirCleanupManagerPurge task started.7:49:47.643 AM INFO DatadirCleanupManagerPurge task completed.7:49:47.646 AM INFO QuorumPeerMainStarting quorum peer7:49:50.747 AM ERROR QuorumPeerMainUnexpected exception, exiting abnormal

Hello, I tried to install a trial CDP Cluster following the given procedure: https://docs.cloudera.com/cdp-private-cloud-base/latest/installation/topics/cdpdc-trial-installation.htmlThe Cloudera Manager used is https://archive.cloudera.com/cm7/7.4.4/cloudera-manager-installer.bin The cluster deployment freezes when Cloudera Manager tries to deploy the agent to the nodes.An unhelpfull error is reported.Digging into this, it fails because passphraseConfirm is empty. But there is no passphraseConfirm field in the form. (See screen capture below) My question is simple: did I miss anything ? How can I complete this level step ? Context: Nodes are AWS instances with Ubuntu 20.04. User connect with a Key. I'm connected to the node through an SSH connection with port redirection of 7180 (this explains the localhost address) Thank you for your help.  

Dear Supporters,I'm newbie at Kudu.I try to run Kudu with Python on Docker from centos/python-36-centos7.I found a rpm's which comes from  https://archive.cloudera.com/kudu/redhat/7/x86_64/kudu/5.12.2/RPMS/x86_64/(kudu 1.4.0) on the other hand, that document says to usehttps://docs.cloudera.com/runtime/7.2.0/kudu-development/topics/kudu-python-client.html http://username:password@archive.cloudera.com/p/cdh7/7.0.3/redhat7/yum/kudu/kudu-1.11.0.7.0.3.0-98.x86_64.rpm In that point I'm very confused. New one need the username and password. How can I get that rpm's? What is the price for using that rpm's?Thanks, 

Hi Everyone, I use ListSFTP and FetchSFTP to collect the files that lines.I want to filter the files based on the third field.I want to collect the files that have the year 1995 only in the lines. |226789|23-Feb-1996|1995|0|1|1|0|0|0|1|0|0|0|0|0|0|0|0|0|0|0|0 |226780|08-Mar-1996|1996|0|0|0|0|0|1|0|0|0|0|0|0|0|0|0|0|0|0|0 |222507|01-Jan-1995|1995|0|0|0|0|0|1|0|0|0|0|0|0|0|0|0|0|0|0|0 |22308|01-Jan-1995|1995|0|1|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0 |222707|01-Jan-1995|1995|0|1|0|0|0|0|0|0|1|0|0|0|0|0|0|0|1|0|0

Hi All,  Currently i'm trying to asses if core component of data platform can run on aws ec2 graviton2.,  Is there reference on the compatiblity of CDP public cloud with the ARM processor? Thanks

The components in HDP 3.1.5 is outdated and lack key security functionality. Grafana is running v6.4.2, but has a major security issue that was patched in future releases: https://grafana.com/blog/2020/06/03/grafana-6.7.4-and-7.0.2-released-with-important-security-fix/Infra Solr is running SOLR 7.7 and haa a RCE vulnerability. This was patched in SOLR 8.3, which is not part of InfraSolr. Zookeeper packaged is 3.4.6, but SSL implementation was add in 3.5.5 I saw some questions talking about "Patch Upgrades" but is there a guide to upgrading individual components in a cluster via Ambari or however? 

Getting out of memory error while running merge query. From Cloudera manager Hive on tez Client Java Heap Size = 8GB, Java Heap Size of HiveServer2 = 16GB Destination table is partitioned by date. Log is given following. Please anyone can give suggestions what configuration is needed to be changed here? ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0INFO : org.apache.tez.common.counters.DAGCounter:INFO : NUM_FAILED_TASKS: 1INFO : NUM_SUCCEEDED_TASKS: 312INFO : TOTAL_LAUNCHED_TASKS: 313INFO : DATA_LOCAL_TASKS: 103INFO : RACK_LOCAL_TASKS: 2INFO : AM_CPU_MILLISECONDS: 20170INFO : AM_GC_TIME_MILLIS: 14INFO : File System Counters:INFO : FILE_BYTES_READ: 1692027391INFO : FILE_BYTES_WRITTEN: 1446300353INFO : HDFS_BYTES_READ: 1490822899INFO : HDFS_BYTES_WRITTEN: 316667034INFO : HDFS_READ_OPS: 46265INFO : HDFS_WRITE_OPS: 813INFO : HDFS_OP_CREATE: 610INFO : HDFS_OP_GET_FILE

Is it safe to delete older ranger audit logs?/ranger/audit/hdfs/20200901/hdfs_ranger_audit_xyz.logas my nodes running out of space. 

Hi all,  I have a scenario where:1. I get a list of flowfiles in Json format2. Using Jolt I extract the fields and send it to an API3. The result of the API needs to find the match in the original json that was ingest in the step one. I've not understood yet what kind of processors I need to implement, I was looking at the LookupRecord, but I haven't so far understood what to do to make the lookup happen in the right flowfile?  Could someone guide me what is necessary to perform it?

Hi,I am getting above error from Hive while trying to query a table.This is coincidentally during an insert operation on the table... does it mean I cant access the table while a hive insert operation is ongoing...? table contains lots of rows partitioned by two columns  

Hi Community, I need you support. I have 3 node-managers and after rebooting 1 node-manager does not start.(RedHat 7.6 / Hadoop 2.6.5 / Ambari 2.5.1.0). Added the full log:https://drive.google.com/file/d/1Osz9nV5PSBAeu8eiiYsJOsib0h3E3Gjk/view?usp=sharing ----------------------------------------------------------FATAL nodemanager.NodeManager (NodeManager.java:initAndStartNodeManager(549)) - Error starting NodeManagerorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: NMWebapps failed to start.at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:116)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:302)at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initA

Hi,We are currently using Python2.7 with HDP3.1.5 (PySpark) and RHEL 7.9.We are planning to upgrade to Python3.x as Python2.7 support has ended.If anyone could please help determining the latest stable Python3.x? version that will best suit or compatible with HDP3.1.5 (Spark2.3) and RHEL 7.9?ORLet me know steps I should follow to determine the best stable Python3.x? version for the HDP stack? Thanks,PB 

Hello,We are observing our Impala Catalogue Server’s process frequently gets exited / killed (This role encountered 1 unexpected exit(s) in the previous 5 minute(s).This included 1 exit(s) due to OutOfMemory errors. Critical threshold: any.)I was going through this article (https://community.cloudera.com/t5/Support-Questions/Cloudera-6-2-1-Impala-GC-Overhead-limit-Exceeded/td-p/293225) it seems to be Heap Memory related, would like to know if there is any way / calculations to find how much Heap should be allocated to avoid these issues. CM / CDH 5.16.2Java Heap Size of Catalog Server in Bytes = 15Gb Appreciate any guidance in this regard.

Hello,I was trying to take report from HDFS Report and get the below messageUpon checking /var/log/cloudera-scm-headlamp and see the following error  021-04-15 18:09:44,934 ERROR com.cloudera.headlamp.HeadlampIndexManager: Index build failed for service hdfsjava.util.concurrent.ExecutionException: java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Unsupported layout version -64 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at com.cloudera.headlamp.AbstractIndexBuilder.buildIndex(AbstractIndexBuilder.java:80) at com.cloudera.headlamp.HeadlampIndex.buildIndex(HeadlampIndex.java:257) at com.cloudera.headlamp.HeadlampIndex.reindex(HeadlampIndex.java:325) at com.cloudera.headlamp.HeadlampIndexManager.reindexIndexes(HeadlampIndexManager.java:240) at com.cloudera.headlamp.HeadlampIndexManager.access$100(HeadlampIndexManager.java

I'm encountering the following error while using beeline.  (Note that I'm able to successfully connect using hive cli. [xxxx@ip-11-1-1-111 data_dir_design]$ beeline -u "jdbc:hive2://ip-11-1-1-111.us-west-1.compute.internal:10000/default;principal=hive/ip-11-1-1-111.us-west-1.compute.internal@EXAMPLE.COM"Connecting to jdbc:hive2://ip-11-1-1-111.us-west-1.compute.internal:10000/default;principal=hive/ip-11-1-1-111.us-west-1.compute.internal@EXAMPLE.COMConnected to: Apache Hive (version 1.2.1000.2.6.5.1175-1)Driver: Hive JDBC (version 1.2.1000.2.6.5.1175-1)Transaction isolation: TRANSACTION_REPEATABLE_READBeeline version 1.2.1000.2.6.5.1175-1 by Apache Hive0: jdbc:hive2://ip-11-1-1-111.us-west-1.compu> show databases;Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [xxxx] does not have [USE] privilege on [Unknown resource!!] (state=42000,code=40000)0: jdbc:hive2://ip-11-1-1-111.us-west-1.compu> When connecting via hive