Q&A

Hi, We have been asked by the client to do a Cloudera Demo. What's the easiest way to deploy such a solution for CDH on a single machine with virtualization? I was used to deploying it using Cloudera Quickstart VM but it seems that no official version of it is downloadable anymore from Cloudera Website. The closest thing looks like to be the Proof-of-concept version but it looks kind of complicated. Not having VM images for the hosts in the cluster is rather upsetting and looks tedious to implement from scratch. Thank you,Cisco

Hi, I've enabled Auto-TLS as instructed in option 2b (https://docs.cloudera.com/cdp-private-cloud-base/7.1.4/security-encrypting-data-in-transit/topics/cm-security-use-case-2.html) but my agents are reporting heartbeat failed in /var/log/cloudera-scm-agent/cloudera-scm-agent.log. Agent at master does not have problems with it's heartbeat. The environment is CDP 7.1.4 with CDP 7.1.3 parcels.I have a development license in place.Also noticed following messages in the certmanager log:"[03/Nov/2020 20:46:58 +0200] 16654 MainThread cert INFO No password file found for host 'masterofanalytics.hemanuniverse.com' at location: /opt/cloudera/AutoTLS/hosts-key-store/masterofanalytics.hemanuniverse.com/cm-auto-host_key.pw. Assuming default in-cluster password."FQDNs and name resolution should be OK.[azureuser@skeletor ~]$ cat /etc/hosts127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4::1 localhost localhost.localdomain localhost6 localhost6.localdomain610.0.0.4

Getting error 500 when connecting - Hue installed from Cloudera Manager. Here are Hue logs:Traceback (most recent call last): File "/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 1228, in communicate req.respond() File "/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 589, in respond self._respond() File "/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/desktop/core/src/desktop/lib/wsgiserver.py", line 601, in _respond response = self.wsgi_app(self.environ, self.start_response) File "/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/core/handlers/wsgi.py", line 157, in __call__ response = self.get_response(request) File "/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/build/env/lib/python2.7/site-packages/Django-1.11.29-py2.7.egg/django/core/handle

Hi all, When we run multiple same request ( not exactly ) we have this exception:   Error: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16] (state=,code=0)   In log file    WARN [HiveServer2-Handler-Pool: Thread-164113]: thrift.ThriftCLIService (:()) - Error fetching results: org.apache.hive.service.cli.HiveSQLException: java.io.IOException: java.lang.RuntimeException: java.util.concurrent.RejectedExecutionException: Task org.apache.hadoop.hbase.client.ResultBoundedCompletionService$QueueingFuture@6cf94617 rejected from java.util.concurrent.ThreadPoolExecutor@458f1611[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 1

Hello Everyone, can you please tell me the difference between DataFrames and DataSets (with examples)? The explanations is still unclearhttp://spark.apache.org/docs/2.4.0/sql-programming-guide.html Thanks, Roshan

I have a processor EvaluateJsonPath processor which is extracting two fields from json file(coming from GETFile processor) which contains{"name": "Jon","title": "king of the nights watch"}after extracting name in EvaluateJsonPath processor as $.name, I want to to write this data to put file processor but before writing I want to change the name of the file such asname+timestamp, ex: jon_23-06-2021 I have tried updateattribute processor something like {name}_${now():format('yyyy-MM-dd-HH-mm-ss')} but its not working, could some one help me here thanks

I have dataflow files which has to be routed on the basis of certain criteria.It has multiple field like ("Customefied_23456: "Null") in the key: value pair. My query is how  to filter out the flowfile which contain the some value for customfield ("customfield_33749": "this is required value",). i.e it does not contain null like  ("Customefied_23456: "Null"). Please let me know how to proceed in this case.thanks in advance.  

Hi All, We are looking for some best practice guidelines on setting limits for users in the limits.conf. We don't use the "*" for username in the limits.conf, they must be defined. CDP is the only thing on our clusters and would like to know what you suggest for each of the service users (i.e. hive/spark/hdfs/oozie, yarn, httpfs, clouder-scm, sqoop, knox, livy, flume, hue, zookeeper, mapred, druid, ranger, solr, etc,  essentially a breakdown of all users that can be added as part of the CDP environment. I have looked through the documentation and haven't found a definitive answer to this question.  Any help deeply appreciated, Thanks in advance,

Hi All, I'm very new to cloudera cdp, I have just been moved over from MapR.I'm looking to find out, i've looked quite a bit but haven't found anything thus far, if there is a cluster wide tool for doing things in the cluster. i.e. in a mapr cluster, if i wanted to get a list of all hdfs volumes with loads of other info on the volumes,  i could type:"maprcli volume list" and it will give me all the below information.  This is just one example, the maprcli interface into MapR has the following commands (below), i'm just wondering if there is anything similar for CDP so that I could work this into home grown scripts that could easily span clusters. Thanks  full list of mapr commands:acerole    acerole [validate] -role [role_name]acl        acl [show|set|edit] -type [cluster|volume] -name <volume name>alarm      alarm [list|raise|clear|clearall|mute|unmute|config|names|group]audit    &nbsp

Hello Team, can you please help me with the connection to impala? [root@rb-hadoop-02 admin]# impala-shell -i <HUE_IP>:27000 --user=hiveStarting Impala Shell without Kerberos authenticationOpened TCP connection to 10.10.104.106:27000Socket error 32: [Errno 32] Broken pipe***********************************************************************************Welcome to the Impala shell.(Impala Shell v3.4.0-SNAPSHOT (134517e) built on Sat Dec 12 11:15:02 UTC 2020)Press TAB twice to see a list of available commands.***********************************************************************************[Not connected] > show dbs;Connection lost, reconnecting...Opened TCP connection to 10.10.104.106:27000Socket error 32: [Errno 32] Broken pipeQuery: show dbsNot connected (use CONNECT to establish a connection)[Not connected] > Thanks, Roshan

Hello Team, can you please advise how I can proceed to:1. Limit the total # of concurrently executing Impala queries system wide2. Limit the memory that an Impala query can use before swapping3. Protect Impala from Yarn contention with Static Service Pools4. Implement Admission Control Queues for different Impala User Groups─ Limit concurrent queries for each queue/pool─ Implement memory limits on the queue/pools and Running Queries5. Put a disk limit on runaway queries specifying the maximum amount of diskstorage, in bytes, that any Impala query can consume on any host on Cloudera? Thanks, Roshan

Hello Team, when executing kafka-topics --list I am getting error kafka-topics --list --bootstrap-server RB-HADOOP-02.mtg.local:9092,RB-HADOOP-03.mtg.local:9092,RB-HADOOP-04.mtg.local:9092 [root@rb-hadoop-02 admin]# kafka-topics --list --bootstrap-server RB-HADOOP-02.mtg.local:9092,RB-HADOOP-03.mtg.local:9092,RB-HADOOP-04.mtg.local:909221/06/11 15:07:40 INFO utils.Log4jControllerRegistration$: Registered kafka:type=kafka.Log4jController MBean21/06/11 15:07:41 INFO admin.AdminClientConfig: AdminClientConfig values:bootstrap.servers = [RB-HADOOP-02.mtg.local:9092, RB-HADOOP-03.mtg.local:9092, RB-HADOOP-04.mtg.local:9092]client.dns.lookup = defaultclient.id =connections.max.idle.ms = 300000metadata.max.age.ms = 300000metric.reporters = []metrics.num.samples = 2metrics.recording.level = INFOmetrics.sample.window.ms = 30000receive.buffer.bytes = 65536reconnect.backoff.max.ms = 1000reconnect.backoff.ms = 50request.timeout.ms = 120000retries = 5retry.backoff.ms = 100sasl.c

Hello,I would like some guidance/ information on data distribution in Kudu T-Servers.We have Kudu cluster of 3 Masters and 9 T-Servers (each t-server has storage of 1TB). We are noticing that space in some t-server is getting consumed rapidly whereas in other its not that much being consumed. Would like to know why this is happening and is there any way this can be overcome, so that data can be distributed evenly across of 9 t-servers.Kudu 1.7.0-cdh5.16.2/ CM 5.16.2 Appreciate any assistance in this regard. ThanksWert 

I installed hadoop-2.7.1 on a centos7 operating system.when I execute the command # hdfs dfs -mkdir /user/Juste . I get an error [hdfs@MASTER ~]$ hdfs dfs -mkdir /user/Juste21/06/20 11:04:04 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicablemkdir: `/user/Juste': No such file or directory

Hi All:I'm new to Apache NiFi. I am able to download and have the apache nifi working perfectly but when it comes to apache NiFi registry, I am getting an error preventing me to start the registry. I ran the batch file after adjusting the port for NiFi registry.run-nifi-registry.bat it gave me an error. I'm guessing some classpath setup issue... not sure C:\Apache nifi-1.12.1\nifi-registry-0.8.0\bin>run-nifi-registry.batcmd.exe /C "C:\Program Files\AdoptOpenJDK\jdk-8.0.242.08-hotspot\bin\java.exe" -cp C:\Apache nifi-1.12.1\nifi-registry-0.8.0\bin\..\conf;C:\Apache nifi-1.12.1\nifi-registry-0.8.0\bin\..\lib\*;C:\Apache nifi-1.12.1\nifi-registry-0.8.0\bin\..\lib\shared\*;C:\Apache nifi-1.12.1\nifi-registry-0.8.0\bin\..\lib\bootstrap\* -Xms512m -Xmx1024m -Dorg.apache.nifi.registry.bootstrap.config.file=C:\Apache nifi-1.12.1\nifi-registry-0.8.0\bin\..\conf\bootstrap.conf org.apache.nifi.registry.NiFiRegistry runError: Could not find or load main class nifi-1.12.1\nifi-reg

Hello Experts,I have an Hortonworks Data Flow cluster managed by Ambari and include the following services: 1- Zookeeper. 2- NIFI. 3- Kafka. 4- Storm.naturlish, that the zookeeper is the spinal cord of the rest of the puzzle pieces, when I tried to start it I got the following error when I started the zookeeper node, and of course, my cluster won't be up until I solve this   I guess because of changing the Horton works binaries public links.  RuntimeError: Failed to execute command '/usr/bin/yum -y install zookeeper_3_4_1_1_4-server', exited with code '1', message: 'http://public-repo-1.hortonworks.com/HDF/centos7/3.x/updates/3.4.1.1/repodata/repomd.xml: [Errno 14] HTTP Error 403 - Forbidden Trying other mirror. can anyone help me with thisAppreciated in advance. 

Hi, the download "https://tiny.cloudera.com/oozie-ext-2.2" is not available anymore.The log is:   sudo: wget --output-document=ext-2.2.zip http://tiny.cloudera.com/oozie-ext-2.2 out: --2021-06-14 09:44:10-- http://tiny.cloudera.com/oozie-ext-2.2 out: Resolving tiny.cloudera.com (tiny.cloudera.com)... 104.236.48.227 out: Connecting to tiny.cloudera.com (tiny.cloudera.com)|104.236.48.227|:80... connected. out: HTTP request sent, awaiting response... 301 Moved Permanently out: Location: https://tinycc.com/tiny/custom_domain_redirect/tiny.cloudera.com/oozie-ext-2.2 [following] out: --2021-06-14 09:44:10-- https://tinycc.com/tiny/custom_domain_redirect/tiny.cloudera.com/oozie-ext-2.2 out: Resolving tinycc.com (tinycc.com)... 159.203.160.179 out: Connecting to tinycc.com (tinycc.com)|159.203.160.179|:443... connected. out: HTTP request sent, awaiting response... 404 Not Found   Although according to the manual, this link is still the valid one.https:

 [mirrormaker-thread-0] Mirror maker thread failure due to java.lang.RuntimeException: { is an invalid regex. at kafka.utils.TopicFilter.<init>(TopicFilter.scala:38) at kafka.utils.Whitelist.<init>(TopicFilter.scala:46) at kafka.tools.MirrorMaker$ConsumerWrapper$$anonfun$init$2.apply(MirrorMaker.scala:316) at kafka.tools.MirrorMaker$ConsumerWrapper$$anonfun$init$2.apply(MirrorMaker.scala:314) at scala.Option.foreach(Option.scala:257) at kafka.tools.MirrorMaker$ConsumerWrapper.init(MirrorMaker.scala:314) at kafka.tools.MirrorMaker$MirrorMakerThread.run(MirrorMaker.scala:215) please help! 

Hi, all, I have faced a problem when I use the hdp 3.1.5 with ambari , because the default hive version 3.1.0 is too high to use the spark 3.1(it's supported with hive version 2.3.7 ), so I need to downgrade the hive version to 2.3.7, how can I do it?

Hello,I have a file in .xlsx format that needs to be transofmerd to .csv format with broken pipe as separator. When I enter broken pipe (¦) as Value Separator in ConvertExcelToCSVProcessor, in CSV file appears "?" as separator instead of "¦". Is there a way to convert .xlsx to .csv with broken pipe delimiter?

Hello, After we upgraded from CDH 5.15 to CDP 7.1.6 runtime. The HUE SAML login got broken. It gives an error below. Any ideas?  ValueError at /saml2/login/Could not deserialize key data.Request Method: GETRequest URL: http://xxxxx.com:8889/saml2/login/?next=/Django Version: 1.11.29Exception Type: ValueErrorException Value:Could not deserialize key data.Exception Location: /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/build/env/lib/python2.7/site-packages/cryptography-2.9-py2.7-linux-x86_64.egg/cryptography/hazmat/backends/openssl/backend.py in _handle_key_loading_error, line 1382Python Executable: /opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/build/env/bin/python2.7Python Version: 2.7.5Python Path:['/opt/cloudera/parcels/CDH-7.1.6-1.cdh7.1.6.p0.10506313/lib/hue/desktop/libs/libsaml/attribute-maps',

Error message:Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1) Following is Details of our cluster HDP Version: 3.1.4Hive Version: 3.1.0.3.1.4.0-315 Configs for Dynamic Partition in hive:hive.exec.dynamic.partition=trueAllow all partitions to be Dynamic=nonstricthive.exec.max.dynamic.partitions=50000hive.exec.max.dynamic.partitions.pernode=20000Sort Partitions Dynamically=falseAllow dynamic numbers of reducers=trueAllow dynamic partition pruning=trueAny help appreciated. 

I am trying to get all documents within a time range using RunMongoAggreation.In console I tried with below query and it worked fine.aggregate([{"$match": {"modified_at": {"$gte": ISODate("2017-02-08T12:10:40.787")}}}])Flowfile has below propertiesstartTimestamp: 2017-02-08T12:10:40.787Below is the query being used in RunMongoAggreation[{"$match": {"modified_at": {"$gte": "${startTimestamp}"}}}]Its not return result

I have some impala queries in a file and I want each time I am running the queries to set the mem_limit = 3gb but in the command not in the file inside impala-shell -f /path/ (and add here the mem_limit) Is this possible?

I use CDH 6.3.2 hadoop is HA I make a workflow with spark in hue run workfolw I get a error             Failing Oozie Launcher, java.net.UnknownHostException: nameservice1 java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1 at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:445) at org.apache.hadoop.hdfs.NameNodeProxiesClient.createProxyWithClientProtocol(NameNodeProxiesClient.java:132) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:351) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:285) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:168) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3237) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:123) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3286) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3254) at org.apach