Q&A

Dear Sir,    May I know if Cloudera also pre-compile winutils.exe for its distribution? We particular interest to the version for CDP 7.1.6. Appreciate.    Thanks and regards, William

I am trying to parse a nested json document using RDD rather than DataFrame. The reason I cannot use DataFrame (the typical code is like spark.read.json) is that the document structure is very complicated. The schema detected by the reader is useless because child nodes at the same level have different schemas. So I try the script below. import jsons='{"key1":{"myid": "123","myname":"test"}}'rdd=sc.parallelize(s).map(json.loads) My next step will be using map transformation to parse json string but I do not know where to start. I tried the script below but it failed. rdd2=rdd.map(lambda j: (j[x]) for x in j) I would appreciate any resource on using RDD transformation to parse json.

We are trying to implement best practices in regards export/import via NiFi Registry. Currently we have one DEV site and multiple operational sites, ideally we would export from DEV and import into our various OPS sites. With that in mind I have a few questions:   What is the best way to provide site-specific configuration for the controllers/Processors? Should we have a unique properties file for each site and we select which one to deploy per site? According to some documentation, later versions of NiFi are planning to remove support for this kind of unique property file loading. Is there a way in the controllers/Processors to provide conditional values based on deployed site? Of particular concern is our Solr "Zookeeper Host List" This is a property value on our Query and Ingest processors, and they currently aren't enabled to support the Variable Registry, so the value is explicitly provided (no variable lookup/replacement). Do we need to set up and use a "Variable Re

Hi, people. I just started using Apache Ranger 1.2.0 with Azure HDInsight Hadoop. Then, I try using Ranger's permission control, but it doesn't work as I want. I know the control works under this flow:https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.0/authorization-ranger/content/apache_ranger_access_conditions.html so that, the deny condition is prior to allow condition. I made the policy which have two conditons; Allow Conditon is the group who includes me and Deny Conditon is just me.And, only the polcy has the access to table A. I guess the group user except me can access(SELECT) the table and I can't, but acutually both can. Is there wrong point? my thought or settings?thanks

On CDP we are using both the Hive (for the Hive Metastores) and Hive on Tez (for the HiveServers). We are getting the below error while trying to run a query based on a condition. I can't share the table information and the exact query but it looks something as below. CREATE EXTERNAL TABLE IF NOT EXISTS XXX ( `1` string, `6` varchar(30), `7` varchar(5), `8` varchar(10)) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'WITH SERDEPROPERTIES('hbase.columns.mapping'=':key, xx:1, xx:5, xx:6')TBLPROPERTIES ( 'hbase.table.name'='YYYYY');  the query looks as follows: select * from XXX where 8 = '1990-10-10';  And we see the below error from the HiveServer [a3ed3b7b-d225-43af-9ac0-76917911a742 HiveServer2-Handler-Pool: Thread-128-EventThread]: Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1d7573cd rejected from java.util.concurrent.ThreadPoolExecutor@194

I have two folders in hdfs with same name under the same directory. I want to get rid of one of them, but how do i do that?

I will use Spark2 in CDP and need to install Python3. Do I need to installation Python3 on every node in the CDP cluster, just only need to install it on one particular node? Spark2 job is executed in JVM containers that could be created on any worker node. I wonder whether the container is created upon a template? If yes, then how the template is created and where is it? Thanks. 

Hello everyone I have development cluster where I lost my ambari database which was on  a mysql database. Is there anyway to reinstall ambari server and have my cluster back up again ?   Thank you for your help

In the cdh6/cdh7 parcels I do not see the lib/hbase/bin/start-hbase.sh script.In cdh5 we used to use hbase-1.2.0-cdh5.16.1/bin/start-hbase.sh to start hbase in pseudo distributed mode.Can someone let me know what is the alternative to using hbase in pseudo distributed mode in CDH6/CDH7 ? Thanks,Sudheendra S.

We have a need to serve our Namenode UI through a proxy and prevent direct access to the UI.   There seems to be 2 ways we could do that but I'm having difficulty in implementing.1.  Set dfs.namenode.http-address to 127.0.0.1 :9870   Unfortunately, it appears that cloudera's implementation does not allow an address in dfs.namenode.http-address, but instead only accept a port number.  This is then combined with the namenode hostname to build the http address.   I attempted to override via a safety valve configuration but that appears to be intercepted as well.2.   Configure the jetty-web.xml with an IPAccessHandler that only allows connections from the local server.   I'm having difficulty finding where in the filesystem I'd set this so that cdh6 will pick this up for the namenode role.This is  Cloudera Enterprise 6.3.3 on Centos 7

I am looking for documentation, examples and other prior art for integrating Ranger logs with IBM QRadar. Our Security team uses QRadar for log filtering and monitoring and requires we send our audit logs to a QRadar event hub for audits.  

When I rerun workflow with ooizeEnvironment detial：CDH6.2.1  

Hello all,I am getting the below error when our application (Java) tries to execute an 'ADD partition' after 'DROP partition IF EXISTS' command in Hive:- """Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. AlreadyExistsException(message:Partition already exists: Partition(values:[xxxx, yyyy, zzzz-zz-zz, tttttttt], dbName:<db_name>, tableName:<tbl_name>"""  Sequence of commands executed:-Thread A: USE <db_name>Thread B: ALTER TABLE <tbl_name> DROP IF EXISTS PARTITION(`i_id` ='xxxx', `c_id` ='yyyy', `dt` ='zzzz-zz-zz', `time` ='tttttttt') PURGEThread C: ALTER TABLE <tbl_name> ADD PARTITION(`i_id` ='xxxx', `c_id` ='yyyy', `dt` ='zzzz-zz-zz', `time` ='tttttttt') Note:-Cluster - 5 Mgr nodes (Hive deployed on 3 of them), 3 Utils and 30 DNsThere are no signs of any latency issues in ambari-server alerts/logs during the tim

On Hadoop cluster, we see intermitted errors in Cloudera Manager diagnostic logs and it fails job running on them. How to investigate it further? we had this issue few weeks back and it became okay but again started since last two days and its intermittent. LoadBalancingKMSClientProvider >>  KMS provider at  threw an IOException:java.io.IOException: Exception while contacting value generator   KMSExceptionsProvider >> User:'hdfs/*******' Method:GET URL:https://************.com:16000/kms/v1/key/haasfba/_eek?num_keys=1&eek_op=generate Response:Internal Server Error-java.io.IOException: Exception while contacting value generatorjava.io.IOException: java.io.IOException: Exception while contacting value generator

Nothing happens when I push download and go there:https://www.cloudera.com/downloads/hortonworks-sandbox/hdp.htmlSome time ago it worked, now I can't see any way to download the sandbox.

CDH 6.3.2 PHOENIX-5.0.0-cdh6.2.0.p0.1308267 add service phoenix   <property><name>hbase.regionserver.wal.codec</name><value>org.apache.hadoop.hbase.regionserver.wal.IndexedWALEditCodec</value></property> <property> <name>phoenix.functions.allowUserDefinedFunctions</name> <value>true</value> <description>enable UDF functions</description></property> adv service and adv client set phoenix.schema.mapSystemTablesToNamespace=true set phoenix.schema.isNamespaceMappingEnabled=true   when I run phoenix-sqlline   hbase region is OutOfMemory   2021-09-09T11:43:01.362+0800: [GC (CMS Initial Mark) [1 CMS-initial-mark: 34175K(34176K)] 48788K(49536K), 0.0309811 secs] [Times: user=0.04 sys=0.01, real=0.03 secs] 2021-09-09T11:43:01.393+0800: [CMS-concurrent-mark-start]2021-09-09T11:43:01.398+0800: [Full GC (Allocation Failure) 2021-09-09T11:43:01.398+0800: [CMS2021-09-09T11:43:01.414+0800

Hi experts,I ran a hive query using tez via beeline to Join tables and got the below error. 2021-09-08T17:07:55,932 INFO [HiveServer2-Background-Pool: Thread-140] hooks.ATSHook: Created ATS Hook2021-09-08T17:07:55,933 INFO [HiveServer2-Background-Pool: Thread-140] ql.Driver: Query ID = hive_20210908170755_9492c1e6-50ee-48da-8353-e49138d8b5272021-09-08T17:07:55,933 INFO [HiveServer2-Background-Pool: Thread-140] ql.Driver: Total jobs = 12021-09-08T17:07:55,933 INFO [HiveServer2-Background-Pool: Thread-140] ql.Driver: Launching Job 1 out of 12021-09-08T17:07:55,933 INFO [HiveServer2-Background-Pool: Thread-140] ql.Driver: Starting task [Stage-1:MAPRED] in serial mode2021-09-08T17:07:55,933 INFO [HiveServer2-Background-Pool: Thread-140] tez.TezSessionPoolManager: QueueName: null nonDefaultUser: false defaultQueuePool: null hasInitialSessions: false2021-09-08T17:07:55,933 INFO [HiveServer2-Background-Pool: Thread-140] tez.TezSessionPoolManager: Created new tez session for queue: null w

Hi,  we have a large number of flowfiles received and processed via NiFi.  Post the NiFi processing we then perform/send the data on via http request. The issue we have at the moment, is that the application (externally to NiFi) can not consume the data fast enough when received from NiFi. What I am looking for is, an ability to only take 1 flowfile every 5 seconds and everything in-between delete/remove.     Have looked at 1 processor that allows for a per flowfile process, however data then starts getting stuck in the queue.  So although processing 1 file every 5 seconds works, its the delete/remove of any other flow files that I need.   Thanks.

We are still running an old version of Clouder, v 5.15.0 (almost at the end of work upgrading) and a few hours ago users reported queries in Hue were not running. CM reported a number of issues so we decided to restart the entire cluster.   Zookeeper will not start, the error seen is below. There is a timeout issue that we've not been able to solve. There were no changes made at the time.    7:49:47.620 AM INFO QuorumPeerConfigReading configuration from: /run/cloudera-scm-agent/process/20394-zookeeper-server/zoo.cfg7:49:47.631 AM INFO QuorumPeerConfigDefaulting to majority quorums7:49:47.634 AM INFO DatadirCleanupManagerautopurge.snapRetainCount set to 37:49:47.634 AM INFO DatadirCleanupManagerautopurge.purgeInterval set to 247:49:47.635 AM INFO DatadirCleanupManagerPurge task started.7:49:47.643 AM INFO DatadirCleanupManagerPurge task completed.7:49:47.646 AM INFO QuorumPeerMainStarting quorum peer7:49:50.747 AM ERROR QuorumPeerMainUnexpected exception, exiting abnormal

Hello, I tried to install a trial CDP Cluster following the given procedure: https://docs.cloudera.com/cdp-private-cloud-base/latest/installation/topics/cdpdc-trial-installation.htmlThe Cloudera Manager used is https://archive.cloudera.com/cm7/7.4.4/cloudera-manager-installer.bin The cluster deployment freezes when Cloudera Manager tries to deploy the agent to the nodes.An unhelpfull error is reported.Digging into this, it fails because passphraseConfirm is empty. But there is no passphraseConfirm field in the form. (See screen capture below) My question is simple: did I miss anything ? How can I complete this level step ? Context: Nodes are AWS instances with Ubuntu 20.04. User connect with a Key. I'm connected to the node through an SSH connection with port redirection of 7180 (this explains the localhost address) Thank you for your help.  

Dear Supporters,I'm newbie at Kudu.I try to run Kudu with Python on Docker from centos/python-36-centos7.I found a rpm's which comes from  https://archive.cloudera.com/kudu/redhat/7/x86_64/kudu/5.12.2/RPMS/x86_64/(kudu 1.4.0) on the other hand, that document says to usehttps://docs.cloudera.com/runtime/7.2.0/kudu-development/topics/kudu-python-client.html http://username:password@archive.cloudera.com/p/cdh7/7.0.3/redhat7/yum/kudu/kudu-1.11.0.7.0.3.0-98.x86_64.rpm In that point I'm very confused. New one need the username and password. How can I get that rpm's? What is the price for using that rpm's?Thanks, 

Hi Everyone, I use ListSFTP and FetchSFTP to collect the files that lines.I want to filter the files based on the third field.I want to collect the files that have the year 1995 only in the lines. |226789|23-Feb-1996|1995|0|1|1|0|0|0|1|0|0|0|0|0|0|0|0|0|0|0|0 |226780|08-Mar-1996|1996|0|0|0|0|0|1|0|0|0|0|0|0|0|0|0|0|0|0|0 |222507|01-Jan-1995|1995|0|0|0|0|0|1|0|0|0|0|0|0|0|0|0|0|0|0|0 |22308|01-Jan-1995|1995|0|1|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0|0 |222707|01-Jan-1995|1995|0|1|0|0|0|0|0|0|1|0|0|0|0|0|0|0|1|0|0

Hi All,  Currently i'm trying to asses if core component of data platform can run on aws ec2 graviton2.,  Is there reference on the compatiblity of CDP public cloud with the ARM processor? Thanks

The components in HDP 3.1.5 is outdated and lack key security functionality. Grafana is running v6.4.2, but has a major security issue that was patched in future releases: https://grafana.com/blog/2020/06/03/grafana-6.7.4-and-7.0.2-released-with-important-security-fix/Infra Solr is running SOLR 7.7 and haa a RCE vulnerability. This was patched in SOLR 8.3, which is not part of InfraSolr. Zookeeper packaged is 3.4.6, but SSL implementation was add in 3.5.5 I saw some questions talking about "Patch Upgrades" but is there a guide to upgrading individual components in a cluster via Ambari or however? 

Getting out of memory error while running merge query. From Cloudera manager Hive on tez Client Java Heap Size = 8GB, Java Heap Size of HiveServer2 = 16GB Destination table is partitioned by date. Log is given following. Please anyone can give suggestions what configuration is needed to be changed here? ERROR : DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0INFO : org.apache.tez.common.counters.DAGCounter:INFO : NUM_FAILED_TASKS: 1INFO : NUM_SUCCEEDED_TASKS: 312INFO : TOTAL_LAUNCHED_TASKS: 313INFO : DATA_LOCAL_TASKS: 103INFO : RACK_LOCAL_TASKS: 2INFO : AM_CPU_MILLISECONDS: 20170INFO : AM_GC_TIME_MILLIS: 14INFO : File System Counters:INFO : FILE_BYTES_READ: 1692027391INFO : FILE_BYTES_WRITTEN: 1446300353INFO : HDFS_BYTES_READ: 1490822899INFO : HDFS_BYTES_WRITTEN: 316667034INFO : HDFS_READ_OPS: 46265INFO : HDFS_WRITE_OPS: 813INFO : HDFS_OP_CREATE: 610INFO : HDFS_OP_GET_FILE