Solved

I am building a 4 node machine and i want to enable hdfs and rm HA. The hdp version is 3.1.0 and has the yarn ats-hbase app always running.  I wanted to know is it mandatory to destroy the app as mentioned https://docs.cloudera.com/HDPDocuments/HDP3/HDP-3.0.1/data-operating-system/content/remove_ats_hbase_before_switching_between_clusters.html before enabling HA? Thanks a lot for your help in advance 

I can't find where could I disable authentication for a Web UI. I'm using HDP 3.1. Any help appreciated!  

hiI created a CentOS 7.7 Linux machine (but also happens with RHEL 7.4) on Azure.I installed the cloudera-manager RPM packages via yumWhen i try to start the cloudera-scm-server-db service I get "permission denied" on /var/lib/cloudera-scm-server-db/dataI added -x flag to the /usr/share/cmf/bin/initialize_embedded_db.sh, and added the "strace" command before the initdb command.I see that the command is running as cloudera-scm user (in the /var/log/messages file)and Is see in the strace output, that it fails on "chmod("/var/lib/cloudera-scm-server-db/data", 0700) = -1 EACCES (Permission denied)"When I run the command from Linux CLI as user cloudera-scm it runs OK.Does the Linux behave differently on Azure (i.e. more security features)I don't think it relates to firewallI see in the /var/log/messages the following messages:Oct 22 12:53:14 cdh-doxi-1 systemd: Created slice User Slice of cloudera-scm.Oct 22 12:53:14 cdh-doxi-1 systemd: Started Session c1 of user cloudera-scm

Hi  Team,   Does kudu tables show up in Cloudera Navigator search for metadata. These tables are created in Impala as Kudu storage type and I'm unable to see them in Navigator. I did not apply any filters in search.   Any possible reason?   Thanks, Meher  

Hello, Friends.   I have a problem with Ranger, I have a Kerberized Cluster, I try to access Ranger UI with user admin:admin and I get error" The username or password you entered is incorrect". likewise changing.   Ranger Admin username for AmbariRanger Admin user's password   Even changing and restarting ambari I can't get in. Could you help me?   Greetings

I am tyring to get this output. Nevertheless some articles points out this is due to a version matter, which I doubt, since I've seen this behavior on multiple environments with different versions is just that I don't know how to enable it. If someone can point me to the right path I would highly appreciate it. Regards!

I would to write a script which will check the status of Hivserver2 is running or not. I would like to connect beeline and show databases. if i get the output then Hiveserver 2 is running. Can any one help in writing the script for this ?

Hi All, I'm trying to submit following Spark2 job on CDH 5.16 cluster and it's only taking first parameter of --packages option and throwing error for second parameter spark2-submit --packages com.databricks:spark-xml_2.11:0.4.1, com.databricks:spark-csv_2.11:1.5.0 /path/to/python-script spark2-submit --packages com.databricks:spark-xml_2.11:0.4.1, com.databricks:spark-csv_2.11:1.5.0 /path/to/python-script.pyException in thread "main" org.apache.spark.SparkException: Cannot load main class from JAR com.databricks:spark-csv_2.11:1.5.0 with URI com.databricks. Please specify a class through --class. at org.apache.spark.deploy.SparkSubmitArguments.error(SparkSubmitArguments.scala:657) at org.apache.spark.deploy.SparkSubmitArguments.loadEnvironmentArguments(SparkSubmitArguments.scala:224) at org.apache.spark.deploy.SparkSubmitArguments.<init>(SparkSubmitArguments.scala:116) at org.apache.spark.deploy.SparkSubmit$$anon$2$$anon$1.<init>(

Hi everyone, I already try to search many topics in internet, especially clouderaforum but can not fix the problem 😞 I am doing: setup a cluster with 2 PCs. /etc/hosts     127.0.0.1 localhost# 127.0.1.1 uvmu02# The following lines are desirable for IPv6 capable hosts::1 ip6-localhost ip6-loopbackfe00::0 ip6-localnetff00::0 ip6-mcastprefixff02::1 ip6-allnodesff02::2 ip6-allrouters# uVMU0X192.168.56.101 uvmu01.uvmu0x.com uvmu01192.168.56.102 uvmu02.uvmu0x.com uvmu02192.168.56.103 uvmu03.uvmu0x.com uvmu03           I already tried re-install ambari-server, ambari-agent, setting public-hostname script, edit hostname info in ambari-agent.ini for each node.           [server]hostname=uvmu01.uvmu0x.com[server]hostname=uvmu02.uvmu0x.com[agent]## uvmu0x## Added following to customize the public hostnamehostname_script=/var/lib/ambari-agent/public_hostname.shpublic_hostname_script=/var/lib/ambari-agent/public_hostname.sh   &n

Hi, I am getting a 500 error when submitting the job.   In oozie log, the error is:   2019-10-17 13:14:49,070 ERROR org.apache.oozie.servlet.V1JobsServlet: SERVER[poc-01.cdh-cluster.internal] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] URL[POST <a href="<a href="http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs" target="_blank">http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs</a>" target="_blank"><a href="http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs</a" target="_blank">http://poc-01.cdh-cluster.internal:11000/oozie/v2/jobs?action=start&user=hdfs</a</a>>] error, java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not foundjava.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem not found at org.apache.hadoop.conf.Config

I installed HDS 3.0 with virtualbox,  when I try to open data analytics studio UI http://sandbox-hdp.hortonworks.com:30800 I got blank page. Please let me know how to fix itThank you very much

I tried to find solution that fits my use case and tried many things, but failed. Please help me. Thank you in advance.   I have input in following format: 12, 2019-07-08 00:02:54.436062+00 23, 2019-07-08 00:48:41.23138+00 ..   how do i create table with timestamp datatype for 2nd column. Please don't suggest regexSerDe. (because this is dummy example, my test string is very complex) In all my input data, all values in 2nd col have '+00' in the end. So, I don't even have to convert to any timezone, i can just skip last 3 characters when parsing. Is there any way to do that? What I have tried   CREATE EXTERNAL TABLE test (   id int,   ts timestamp) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'WITH SERDEPROPERTIES ('serialization.format' = ',','field.delim' = ',') LOCATION 's3://user/'  

Hi, guys My master's /usr/bin was deleted, and i have copy other slave's /usr/bin to the master. but when i type spark-shell, it occur this error: spark-shell error but when i type /opt/cloudera/parcels/CDH/bin/spark-shell, it works fine. Seems the commands in /user/bin directory should be soft link to/ opt/cloudera/parcels/CDH , is there any way to recover it?   thanks a lot.

I am trying search my host for CDH cluster installation. I search using IP and hostname. I can not found cloudera host.  Here log from   /var/log/cloudera-scm-server/cloudera-scm-server.log       2019-10-15 16:46:54,028 INFO 1759898621@scm-web-9:com.cloudera.server.cmf.node.NodeScannerService: Request 2 returning 0/1 scans2019-10-15 16:46:56,027 INFO 1759898621@scm-web-9:com.cloudera.server.cmf.node.NodeScannerService: Request 2 returning 0/1 scans2019-10-15 16:46:58,042 INFO 1759898621@scm-web-9:com.cloudera.server.cmf.node.NodeScannerService: Request 2 returning 0/1 scans2019-10-15 16:47:00,037 INFO 1759898621@scm-web-9:com.cloudera.server.cmf.node.NodeScannerService: Request 2 returning 0/1 scans2019-10-15 16:47:02,040 INFO 1759898621@scm-web-9:com.cloudera.server.cmf.node.NodeScannerService: Request 2 returning 0/1 scans2019-10-15 16:47:04,040 INFO 1759898621@scm-web-9:com.cloudera.server.cmf.node.NodeScannerService: Request 2 returning 0/1 scans2019-10-15

Hi, we have multiple clusters which are managed by different instances of Cloudera Manager. It would be helpful to compare the complete configuration from the different cluster, i.e. to find out how the monitoring tresholds of service X is set in cluster A and cluster B. I have already managed to receive the complete JSON config (api/v10/cm/deployment), but it does not help much to compare them (even with a specific JSON comparison tool), because it really is a complex file. Does anyone has a good idea or some experience for this scenario? Thanks a lot, Silvan

I want to setup the Edge node for HDP 3.1 so i need your help. Please share the steps to build it.

I'm struggling with enabling GPU Support on our CDSW 1.6. We have a CDSW Cluster with 1 Master-Node and 2 Woker-Nodes (one of the Worker-Nodes is equipped with NVIDIA GPUs)What I did so far:Successfully upgraded our CDSW Cluster to CDSW 1.6Prepared the GPU-Host regarding this CDSW NVIDIA-GPU GuideDisabled Nouveau on the GPU-HostInstalled successfully the NVIDIA Driver with respect to the right kernel header version.(This NVIDIA-Guide was very useful)Checked that the NVIDIA driver is correct installed with the command nvidia-smiThe output shows the correct number of NVIDIA GPUs which are installed on the GPU-Host. Added the GPU-Host with Cloudera-Manager to the CDSW-Cluster and added the worker and docker daemon role instances to the GPU-Host.Enable GPU Support for CDSW on Cloudera-Manager.Restarted the CDSW Service with Cloudera-Manager successfully.When I log into the CDSW Web-GUI I don't see any available GPUs as shown in the CDSW NVIDIA-GPU Guide.B

Hi, I have a Nifi workflow that puts data from MS SQL to Oracle.  I have some queries that I used " binding" such as    ... where MUST.MID = ${MID} and not exists (select a_id from tbl_a where a_id = ${MID}).   But the variables behaves like "literal" variables on Oracle. (it runs everytime such as  where MUST.MID = 50 and not exists (select a_id from tbl_a where a_id = 50))   What can I do for binding in Oracle part?   Thank you in advance. 

We're on CDM 5.14.2 with CDH 5.13.3 Both Spark 1.6 and Spark 2.3.3 are installed (some apps are still using Spark 1.6, can't remove it yet) Now when I'm starting pyspark with config file for Spark2, it still runs pyspark with Spark 1.6 e.g. pyspark --properties-file /etc/spark2/conf/spark-defaults.conf it shows after the ASCII Spark logo: version 1.6.0 using verbose mode it shows the paths are pointing to Spark 2 spark.yarn.jars,local:/opt/cloudera/parcels/SPARK2-2.3.0.cloudera3-1.cdh5.13.3.p0.458809/lib/spark2/jars/*   Why is pyspark still referring to Spark 1.6 ? How can I force it to use spark 2.3.3 ?  

Hi,   I would like to help me to optimize my cluster (3 Servers) with kafka, I understand that the default configuration is valid but I want to improve it. for example the topic of log cleaning. I saw several forums and different criteria but if you can give me a tip, I will be grateful.   Greetings

Searching on internet I found that I can clean the tmp directory without problem on a cloudera cluster, so I did it. Now, when I try to start the cluster all services goes up well, but yarn failed to start with this error:   19/10/16 15:50:17 FATAL nodemanager.NodeManager: Error starting NodeManagerorg.apache.hadoop.yarn.exceptions.YarnRuntimeException: NMWebapps failed to start. at org.apache.hadoop.yarn.server.nodemanager.webapp.WebServer.serviceStart(WebServer.java:79) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceStart(NodeManager.java:329) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:563) at org.apache.hadoop.yarn.server.n

we have a beeline process that inserts data into a hive table we can only access the table via impala views so when the beeline job is done, we want to refresh the tables and select rows from the table via impala views could we write a user defined function that 1. refreshes the table in impala 2. selects * from the refreshed table   call that UDF from an impala view

Hello All   Is it possible to install Atlas 2.0 in HDP 2.6.5 or do we need HDP 3.0? Also, Can you please let me know the latest version of Atlas that can be installed in HDP 2.6.5?   Thanks

I need clarification about Metastore URI configuration of Hive data server. The problem is that in the official cloudera documentation you can see that it is a requirement to block external applications accessing Hive Metastore in order to secure cluster with Sentry:   https://www.cloudera.com/documentation/enterprise/latest/topics/sg_sentry_service_config.html#concept_z5b_42s_p4__section_lvc_4g4_rp   Block the external applications from accessing the Hive metastore: In the Cloudera Manager Admin Console, select the Hive service. On the Hive service page, click the Configuration tab. In the search well on the right half of the Configuration page, search for Hive Metastore Access Control and Proxy User Groups Override to locate the hadoop.proxyuser.hive.groups parameter and click the plus sign. Enter hive into the text box and click the plus sign again. Enter hue into the text box. Enter sentry into the text box. Click Save Changes.   ODI is also an external application

nifi Json data using routeonattributeto to split a...  I am currently working to consume data using Nifi related to tealium event stream and load into HDFS. Need help in filtering the data when source misses to send data for json attribute.   {"account":"newt","twitter:description":"Discover when your favorite New TV shows and hosts are being shown. ","og:locale":"en_US","dcterms:publisher":"New TV","original-source":"www.newtv.com/","og:url":"www.newtv.com/show/program-guide"}},"post_time":"2019-10-09 11:27:46","useragent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.90 Safari/537.36","event_id":"12345"}   Sample above message. I am currently stuck with filtering the data when source misses to send data for event_id attribute.   Current Nifi flow, Consume Kafka -> Evaluate Json Path -> Jolttransform Json -> Evaluate Json Path-> RouteOnAttribute -> Merge Content -> Evaluate Jso