Q&A

Hi everyone, I already try to search many topics in internet, especially clouderaforum but can not fix the problem 😞 I am doing: setup a cluster with 2 PCs. /etc/hosts     127.0.0.1 localhost# 127.0.1.1 uvmu02# The following lines are desirable for IPv6 capable hosts::1 ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allrouters# uVMU0X192.168.56.101 uvmu01.uvmu0x.com uvmu01192.168.56.102 uvmu02.uvmu0x.com uvmu02192.168.56.103 uvmu03.uvmu0x.com uvmu03           I already tried re-install ambari-server, ambari-agent, setting public-hostname script, edit hostname info in ambari-agent.ini for each node.           [server]hostname=uvmu01.uvmu0x.com[server]hostname=uvmu02.uvmu0x.com[agent]## uvmu0x## Added following to customize the public hostnamehostname_script=/var/lib/ambari-agent/public_hostname.shpublic_hostname_script=/var/lib/ambari-agent/public_hostname.sh   &n

Hi, I am getting a 500 error when submitting the job.   In oozie log, the error is:   2019-10-17 13:14:49,070 ERROR org.apache.oozie.servlet.V1JobsServlet: SERVER[poc-01.cdh-cluster.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST <a href="<a href="http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs" target="_blank">http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs</a>" target="_blank"><a href="http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs</a" target="_blank">http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs</a</a>>] error, java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not foundjava.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found at org.apache.hadoop.conf.Config

I installed HDS 3.0 with virtualbox,  when I try to open data analytics studio UI http://sandbox-hdp.hortonworks.com:30800 I got blank page. Please let me know how to fix itThank you very much

I tried to find solution that fits my use case and tried many things, but failed. Please help me. Thank you in advance.   I have input in following format: 12, 2019-07-08 00:02:54.436062+00 23, 2019-07-08 00:48:41.23138+00 ..   how do i create table with timestamp datatype for 2nd column. Please don't suggest regexSerDe. (because this is dummy example, my test string is very complex) In all my input data, all values in 2nd col have '+00' in the end. So, I don't even have to convert to any timezone, i can just skip last 3 characters when parsing. Is there any way to do that? What I have tried   CREATE EXTERNAL TABLE test (   id int,   ts timestamp) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'WITH SERDEPROPERTIES ('serialization.format' = ',','field.delim' = ',') LOCATION 's3://user/'  

Hi, guys My master's /usr/bin was deleted, and i have copy other slave's /usr/bin to the master. but when i type spark-shell, it occur this error: spark-shell error but when i type /opt/cloudera/parcels/CDH/bin/spark-shell, it works fine. Seems the commands in /user/bin directory should be soft link to/ opt/cloudera/parcels/CDH , is there any way to recover it?   thanks a lot.

I am trying search my host for CDH cluster installation. I search using IP and hostname. I can not found cloudera host.  Here log from   /var/log/cloudera-scm-server/cloudera-scm-server.log       2019-10-15 16:46:54,028 INFO 1759898621@scm-web-9:com.cloudera.server.cmf.node.NodeScannerService: Request 2 returning 0/1 scans2019-10-15 16:46:56,027 INFO 1759898621@scm-web-9:com.cloudera.server.cmf.node.NodeScannerService: Request 2 returning 0/1 scans2019-10-15 16:46:58,042 INFO 1759898621@scm-web-9:com.cloudera.server.cmf.node.NodeScannerService: Request 2 returning 0/1 scans2019-10-15 16:47:00,037 INFO 1759898621@scm-web-9:com.cloudera.server.cmf.node.NodeScannerService: Request 2 returning 0/1 scans2019-10-15 16:47:02,040 INFO 1759898621@scm-web-9:com.cloudera.server.cmf.node.NodeScannerService: Request 2 returning 0/1 scans2019-10-15 16:47:04,040 INFO 1759898621@scm-web-9:com.cloudera.server.cmf.node.NodeScannerService: Request 2 returning 0/1 scans2019-10-15

Hi, we have multiple clusters which are managed by different instances of Cloudera Manager. It would be helpful to compare the complete configuration from the different cluster, i.e. to find out how the monitoring tresholds of service X is set in cluster A and cluster B. I have already managed to receive the complete JSON config (api/v10/cm/deployment), but it does not help much to compare them (even with a specific JSON comparison tool), because it really is a complex file. Does anyone has a good idea or some experience for this scenario? Thanks a lot, Silvan

I want to setup the Edge node for HDP 3.1 so i need your help. Please share the steps to build it.

I'm struggling with enabling GPU Support on our CDSW 1.6. We have a CDSW Cluster with 1 Master-Node and 2 Woker-Nodes (one of the Worker-Nodes is equipped with NVIDIA GPUs)What I did so far:Successfully upgraded our CDSW Cluster to CDSW 1.6Prepared the GPU-Host regarding this CDSW NVIDIA-GPU GuideDisabled Nouveau on the GPU-HostInstalled successfully the NVIDIA Driver with respect to the right kernel header version.(This NVIDIA-Guide was very useful)Checked that the NVIDIA driver is correct installed with the command nvidia-smiThe output shows the correct number of NVIDIA GPUs which are installed on the GPU-Host. Added the GPU-Host with Cloudera-Manager to the CDSW-Cluster and added the worker and docker daemon role instances to the GPU-Host.Enable GPU Support for CDSW on Cloudera-Manager.Restarted the CDSW Service with Cloudera-Manager successfully.When I log into the CDSW Web-GUI I don't see any available GPUs as shown in the CDSW NVIDIA-GPU Guide.B

Hi, I have a Nifi workflow that puts data from MS SQL to Oracle.  I have some queries that I used " binding" such as    ... where MUST.MID = ${MID} and not exists (select a_id from tbl_a where a_id = ${MID}).   But the variables behaves like "literal" variables on Oracle. (it runs everytime such as  where MUST.MID = 50 and not exists (select a_id from tbl_a where a_id = 50))   What can I do for binding in Oracle part?   Thank you in advance. 

We're on CDM 5.14.2 with CDH 5.13.3 Both Spark 1.6 and Spark 2.3.3 are installed (some apps are still using Spark 1.6, can't remove it yet) Now when I'm starting pyspark with config file for Spark2, it still runs pyspark with Spark 1.6 e.g. pyspark --properties-file /etc/spark2/conf/spark-defaults.conf it shows after the ASCII Spark logo: version 1.6.0 using verbose mode it shows the paths are pointing to Spark 2 spark.yarn.jars,local:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/jars/*   Why is pyspark still referring to Spark 1.6 ? How can I force it to use spark 2.3.3 ?  

Hi,   I would like to help me to optimize my cluster (3 Servers) with kafka, I understand that the default configuration is valid but I want to improve it. for example the topic of log cleaning. I saw several forums and different criteria but if you can give me a tip, I will be grateful.   Greetings

Searching on internet I found that I can clean the tmp directory without problem on a cloudera cluster, so I did it. Now, when I try to start the cluster all services goes up well, but yarn failed to start with this error:   19/10/16 15:50:17 FATAL nodemanager.NodeManager: Error starting NodeManagerorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: NMWebapps failed to start. at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:79) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:329) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:563) at org.apache.hadoop.yarn.server.n

we have a beeline process that inserts data into a hive table we can only access the table via impala views so when the beeline job is done, we want to refresh the tables and select rows from the table via impala views could we write a user defined function that 1. refreshes the table in impala 2. selects * from the refreshed table   call that UDF from an impala view

Hello All   Is it possible to install Atlas 2.0 in HDP 2.6.5 or do we need HDP 3.0? Also, Can you please let me know the latest version of Atlas that can be installed in HDP 2.6.5?   Thanks

I need clarification about Metastore URI configuration of Hive data server. The problem is that in the official cloudera documentation you can see that it is a requirement to block external applications accessing Hive Metastore in order to secure cluster with Sentry:   https://www.cloudera.com/documentation/enterprise/latest/topics/sg_sentry_service_config.html#concept_z5b_42s_p4__section_lvc_4g4_rp   Block the external applications from accessing the Hive metastore: In the Cloudera Manager Admin Console, select the Hive service. On the Hive service page, click the Configuration tab. In the search well on the right half of the Configuration page, search for Hive Metastore Access Control and Proxy User Groups Override to locate the hadoop.proxyuser.hive.groups parameter and click the plus sign. Enter hive into the text box and click the plus sign again. Enter hue into the text box. Enter sentry into the text box. Click Save Changes.   ODI is also an external application

nifi Json data using routeonattributeto to split a...  I am currently working to consume data using Nifi related to tealium event stream and load into HDFS. Need help in filtering the data when source misses to send data for json attribute.   {"account":"newt","twitter:description":"Discover when your favorite New TV shows and hosts are being shown. ","og:locale":"en_US","dcterms:publisher":"New TV","original-source":"www.newtv.com/","og:url":"www.newtv.com/show/program-guide"}},"post_time":"2019-10-09 11:27:46","useragent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36","event_id":"12345"}   Sample above message. I am currently stuck with filtering the data when source misses to send data for event_id attribute.   Current Nifi flow, Consume Kafka -> Evaluate Json Path -> Jolttransform Json -> Evaluate Json Path-> RouteOnAttribute -> Merge Content -> Evaluate Jso

Hive's execution engine supports MapReduce & Tez. Does it supports Spark in HDP 3.x. https://issues.apache.org/jira/browse/HIVE-7292

Hi, I have an issue with distcp authentication between a kerberos secured cluster (HDP2.6.1.0-129) and an unsecured cluster. (HDP3.0.1.0-187) The compatibility matrix for the versions says they should interoperate I am running the following from the secure cluster:   hadoop distcp -D ipc.client.fallback-to-simple-auth-allowed=true swebhdfs://FQDN(secure cluster):50470/tmp/test.sh swebhdfs://FQDN(insecure cluster):50470/tmp/test.sh   Both ends have TLS enabled.I have a truststore configured and working.  I can see packets outbound from the client and arriving at the server using tcpdump   The error returned by the distcp command above is: 19/10/15 09:58:48 ERROR tools.DistCp: Exception encounteredorg.apache.hadoop.security.AccessControlException: Authentication requiredat org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:460)at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:114)at org.apache.hadoop

I installed successfully CDSW 1.6 on 1 Master-Node and 2 Worker-Nodes. Unfortunately I get on both of the Worker-Nodes these error messages when I check the nodes with cdsw status:(On the Master-Node there are no error messages) Failed to run CDSW secrets check.Failed to run CDSW persistent volumes check.Failed to run CDSW persistent volumes claims check.Failed to run CDSW Ingresses check.Checking web at url: http://workbench.company.comOK: HTTP port checkCloudera Data Science Workbench is not ready yetI couldn't see any drawbacks till this time and everything seems working quite nice. But I'm interested why I get these error messages and what they mean.Can someone help please?Thanks!

Hello, I am using Sandbox HDP 3.0, I am having a Datanote error not running (I started it up, it did not report anything but then stopped immediately)I have attached the image file and the log file. Thanks to everyone for watching and helping me fix the mistake. Thank you very much!   File log ************************************************************/2019-10-13 09:56:41,017 INFO datanode.DataNode (LogAdapter.java:info(51)) - STARTUP_MSG: /************************************************************STARTUP_MSG: Starting DataNodeSTARTUP_MSG: host = sandbox-hdp.hortonworks.com/172.18.0.2STARTUP_MSG: args = []STARTUP_MSG: version = 3.1.1.3.0.1.0-187STARTUP_MSG: classpath = /usr/hdp/3.0.1.0-187/hadoop/conf:/usr/hdp/3.0.1.0-187/hadoop/lib/jsp-api-2.1.jar:/usr/hdp/3.0.1.0-187/hadoop/lib/jetty-servlet-9.3.19.v20170502.jar:/usr/hdp/3.0.1.0-187/hadoop/lib/avro-1.7.7.jar:/usr/hdp/3.0.1.0-187/hadoop/lib/jcip-annotations-1.0-1.jar:/usr/hdp/3.0.1.0-187/hadoop/lib/stax2-api-3.1.4.jar:/usr/hd

i am trying connected mysql java connector to my mariadb using this docs https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/prepare_cm_database.html. I am getting this error messege       [root@cloudera java-1.6.0]# /usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm my_passwordJAVA_HOME=/usr/java/jdk1.6.0_31Verifying that we can write to /etc/cloudera-scm-serverCreating SCM configuration file in /etc/cloudera-scm-serverExecuting: /usr/java/jdk1.6.0_31/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/usr/share/cmf/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db.Exception in thread "main" java.lang.UnsupportedClassVersionError: org/postgresql/util/PSQLException : Unsupported major.minor version 51.0 at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.d

We recently upgraded from hdp 2.6.4 to 2.6.5. After the upgrade i am not able to execute this command -  show partitions `<database_name>.<table_name>`   The command works fine with 2.6.4 version. Any help would be appreciated. Thanks!

Good Morning.I am doing a small example with HIVE and MANAGED TABLE. My environment is Oracle VM virtual Box 6.0Hive 1.1.0-cdh5.13.0 I indicate the steps I have taken:1st)CREATE DATABASE scenariox 2nd) USE scenariox CREATE TABLE scenariox(smallint order;name string;)ROW FORMAT DELIMITED FIELDS TERMINATED BY ','; 3rd) I load the table with my csv file that is stored in HDFS HDFS> /scenariox/scenariox.csv LOAD DATA LOCAL INPATH '/scenariox/scenariox.csv' INTO TABLE scenariox The table loads correctly. I can run: '' Select * from scenariox ''. I recover data correctly. The table has been loaded successfully.The directories in hive are created correctly: /user/hive/warehouse/scenariox.db/scenariox/scenariox.csv I execute the sentence: SELECT COUNT (1) FROM scenariox;MAP / REDUCE launched   My problem is: I can't find the output file that produces the map / reduce. What directory is it stored in? I thought that a file of type '00000_0' was generated and stored in the HDFS. Can some

I am trying install cloudera 5.16.2(single mode). In cluster instalasion  i am get this error.   Spoiler (Highlight to read) Spoiler (Highlight to read) /tmp/scm_prepare_node.ZzLnmqNEusing SSH_CLIENT to get the SCM hostname: 10.0.0.4 34302 22opening logging file descriptorStarting installation script...Acquiring installation lock...BEGIN flock 4END (0)Detecting root privileges...effective UID is 0Detecting distribution...BEGIN grep Tikanga /etc/redhat-releaseEND (1)BEGIN grep 'CentOS release 5' /etc/redhat-releaseEND (1)BEGIN grep 'Scientific Linux release 5' /etc/redhat-releaseEND (1)BEGIN grep Santiago /etc/redhat-releaseEND (1)BEGIN grep 'CentOS Linux release 6' /etc/redhat-releaseEND (1)BEGIN grep 'CentOS release 6' /etc/redhat-releaseEND (1)BEGIN grep 'Scientific Linux release 6' /etc/redhat-releaseEND (1)BEGIN grep Maipo /etc/redhat-releaseEND (1)BEGIN grep 'CentOS Linux release 7' /etc/redhat-releaseCentOS Linux release 7.5.1804 (Core)END (0)/etc/redhat-release ==>