Q&A

Hi Team,We upgraded our spark to 2.3.0.2.6.5.0-292. When I export the spark_major_version=2, and if I type spark-shell then it will be connecting to 2.3.0.2.6.5.0-292. I want to use spark 2.1.0.2.6.0.23-2 version, how can I do that. Please help us on this.

I have a Cloudera CDH cluster currently running on Oracle JDK, and I need to know the proper process to migrate the entire cluster over to OpenJDK.

Hi Team, Greetings! There was a need to move the kudu installation from my current cluster of AWS machines to new cluster of  AWS Machines.So I did the following steps.Took the back/sanpshot of disk haivng the kudu data Directory.Installed CDH and kudu on the new AWS cluster and attached the Disk from the older cluster to the newer one.After the installation I checked for kudu health in Cloudera Manager, kudu cluster was healthy and all new tablet servers were running fine along with master.when I ran the ksck command all except few the tables were in CONSENSUS_MISMATCH stateLogs suggest the that master is still trying to refer to the tablet servers which is having IP of older machines.Can anybody help in pointing out why this is happening.  RegardsParth 

How can I trigger sample Apache Spark and PySpark jobs using Apache Oozie?

Changes that i tried but did not help:set home directory for javachanged the warehouse root directorycreated the znode for hiveserver2 manually (but it returns empty set)reinstalled zookeeper and hive againstderr: /var/lib/ambari-agent/data/errors-252.txtTraceback (most recent call last): File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 143, in <module> HiveServer().execute() File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute method(env) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_server.py", line 53, in start hive_service('hiveserver2', action = 'start', upgrade_type=upgrade_type) File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/HIVE/package/scripts/hive_service.py", line 101, in hive_service wait_for_znode() File "/usr/lib/ambari-agent/lib/resource_management/libraries/functions/decorator.py", line 54, in wrappe

Hi , We are using 12 Datanodes in our cluster. And we have 3 replication factor. Currently we are doing cost optimization and our total cluster is 35% used. We want to remove one datanode from our cluster, but we want to see first how our cluster performs on 11 datanodes. if it doesn't perform well, we need to restart that datanode again.To experiment this- Do I need to Stop the datanode or Decomission datanode?Please need suggestion.

I am fetching data from pastel(pervasive) database.But every other day, i used to get error--->LNA session timeoutThen i have to go the connection setting and when i disable the connection and then enable it, everything works fine.Is there any way that i can automate the process of disable then enable the connection.

Hi All, does the "Balancer" role itself use any resources?  Does it matter which host the role resides (e.g. NameNode, Edge Node, etc)? Thank you.

hello folks , the nodeManager has suddently stopped in a instance (while stille running for other nodes/intances ).so when I try to start/restart it -via cloudera manager - , an error is shown in the first step : Failed to start role.and I'm using CentOS release 6.10 (Final)please what do you suggest me to look or check in order to resolve this problem ?here's my stdout log : Tue Apr 23 10:18:56 PDT 2019JAVA_HOME=/usr/java/jdk.1.8.0_144using /usr/java/jdk.1.8.0_144 as JAVA_HOMEusing 5 as CDH_VERSIONusing /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop-yarn as CDH_YARN_HOMEusing /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop-mapreduce as CDH_MR2_HOMEusing /var/run/cloudera-scm-agent/process/23960-yarn-NODEMANAGER as CONF_DIRCONF_DIR=/var/run/cloudera-scm-agent/process/23960-yarn-NODEMANAGERCMF_CONF_DIR=/etc/cloudera-scm-agent## A fatal error has been detected by the Java Runtime Environment:## SIGBUS (0x7) at pc=0x00007f8c1fde51a1, pid=300

Installed kafka broker in a node using ambari blueprint with hdp 2.6.5 with ranger-kafka plugin enabled. Created topics using kafka-topics.sh command. Getting TOPIC_AUTHORIZATION_FAILED warning and unable to produce messages. By going through this document (https://cwiki.apache.org/confluence/display/RANGER/Kafka+Plugin#KafkaPlugin-Authorizingtopiccreation ), got to know that ranger-kafka plugin authorization works only for autocreated topics. Tried the same producer command by auto-created topics but still getting the same error. Could anyone please help me to resolve this.

Hi,I would like to use the yarn node label functionality. For this I need to create a yarn node label as user yarn.When I try 'su yarn' it is asking me for a password.How is it possible to change the yarn password within HDP ?Br,Dennis

We have Hadoop cluster based on ambari Since thrift server have poor performance , we decided to replace it with presto Our current Hadoop cluster have the following machines 960 data node machines ( based on redhat 7 OS )Few words about the presto- Presto (or PrestoDB) is an open source, distributed SQL query engine, designed from the ground up for fast analytic queries against data of any size. It supports both non-relational sources, such as the Hadoop Distributed File System (HDFS), We installed the new presto server as the following First we installed the OS ( redhat 7 ) , total 13 machines 1 machine for the presto coordinator And 12 machines for presto workersAfter installing the OS We installed successfully the presto ( presto coordinator + presto workers ) Now we are stuck about how to do the integration between presto cluster to the Hadoop clusterI will give short example about hive connector ( hive.properties )we have the following variable hive.config.resources=/etc/hado

Hi all,I had built up a HDF 3.3.1.0 with ambari, but I got NoClassDefFoundError when I'm building a data processing kafka to druid by Streaming Analytics Manager, how to fix it?My environment:Storm:1.2.1SAM: 0.6.0Kafka: 1.0.0.3.1Druid: 0.12.1Below is the logs in storm worker.java.lang.NoClassDefFoundError: com/fasterxml/jackson/core/JsonProcessingExceptionat org.apache.storm.kafka.spout.KafkaSpout.open(KafkaSpout.java:142) ~[dep-org.apache.storm-storm-kafka-client-jar-1.2.1.3.3.1.0-10.jar.1555487788000:1.2.1.3.3.1.0-10]at org.apache.storm.daemon.executor$fn__10149$fn__10164.invoke(executor.clj:615) ~[storm-core-1.2.1.3.1.0.0-78.jar:1.2.1.3.1.0.0-78]at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:482) [storm-core-1.2.1.3.1.0.0-78.jar:1.2.1.3.1.0.0-78]at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]Caused by: java.lang.ClassNotFoundException: com.fasterxml.jackson.core.JsonProcessingExceptionat java.net.URLClass

When i am running below select unix_timestamp('20190131 05:00:00',"yyyyMMdd HH:mm:ss"); output is 1548910800  but when i am running same thing on a table on a column which is big int select min(IPID.BK_EFF_STRT_DT) ,max(IPID.BK_EFF_STRT_DT) from  ipid; o/p is below +--------------------------+--------------------------+| min(ipid.bk_eff_strt_dt) | max(ipid.bk_eff_strt_dt) |+--------------------------+--------------------------+| 1512709200000 | 1548910800000 why is extra 0 added in 2nd case Also select cast(1548910800000 div 1000 as timestamp) + interval (1548910800000 % 1000) milliseconds; returns : 2019-01-31 05:00:00 This is causing issue when i am trying something like below it returns 0 rows: select count(*) from ipidwhere unix_timestamp('20190124',"yyyyMMdd") between IPID.BK_EFF_STRT_DT AND IPID.BK_EFF_END_DT;

When I try to add a GCP credential in the UI I get this error:Failed to verify the credential: Could not verify credential! 403 ForbiddenOn console I get this:# cb credential create from-file --cli-input-json credential.json --name cloudbreakkeyEnter Password: ERROR: status code: 400, message: Failed to verify the credential: Could not verify credential! 403 Forbiddencb doctor:# cbd doctor===> Deployer doctor: Checks your environment, and reports a diagnose.uname: Linux cbd-deployment-vm 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linuxlocal version:2.9.0latest release:2.9.0docker command exists: OKdocker client version: 1.13.1docker client version: 1.13.1ping 8.8.8.8 on host: OKping github.com on host: OKping 8.8.8.8 in container: [ERROR]ping github.com in container: [ERROR] I assume there must be some networking issue but don't know what to look forEDIT: In the logs I get this stacktrace:com.google.api.client.googleapis.json.GoogleJs

I created a table using Regex Serde in Hive (shown below). In Heu it returns that the creation of the table was successful. However, when I try to return the table (SELECT * FROM pricefile_edited) or view the table in heu, it doesn't work and I get the Error (also below).  The data is a 130 character (each line) with no delimiter. Does anyone know what seems to be the problem, and help? Thanks SELECT * FROM pricefile_edited;CREATE EXTERNAL TABLE pricefile_edited(field1 STRING,field2 STRING,field3 STRING,field4 STRING,field5 STRING,field6 STRING, field7 STRING,field8 STRING,field9 STRING,field10 STRING,field11 STRING,field12 STRING,field13 STRING,field14 STRING,field15 STRING,field16 STRING,field17 STRING,field18 STRING)ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ("input.regex" = "(\\.{12})(\\.{1})(\\.{1})(\\.{24})(\\.{6})(\\.{6})(\\.{13})(\\.{6})(\\.{1})(\\.{4})(\\.{1})(\\.{3})(\\.{17})(\\.{9})(\\.{12})(\\.{1})(\\.{1})(\\.{12})")LOCATION '/use

I'm currently building a customed cdh6.1.0 docker image.Based on centos7:latest.Installed net-tools, openssh-server, openssh-clients and etc.The privileged is set to true to allow systemd boot up.All is fine(though cost me a week to get through) and now I've got images: 1. image installed cloudera-scm-server (used as cloudera manager)2. image installed dnsmasqude to provide dns resolution instead of editing the /etc/hosts (used as dns)3. image loaded with latest cdh6.1.0/cm6.1.0/gplextra6.1.0 repos serving on port 8900 (used as local repo)4. image installed only sshd and ntpd services (used for deployment, run under privileged mode) Then I started a test cluster to deploy CDH on some replicates of the 4th images. Though there were a lot of errors bursted into logs (such as some file cannot be found which in fact exists and well written). After a short duration, the installation completes, the detail should the installation script completed. However the status showed that

I have the following example JSON in a flowfile (data has been randomly imputed for this example) :{"TrackingRequestFirstIp":"192.10.01.01","TrackingRequestLastIp":"100.120.123.456","TrackingRequestCreationTime":"05:49:55","TrackingRequestCreationDate":"20171013","TrackingRequestTrackingNumber":"12349876"}Here is my AVRO schema:{  "name": "IpReports",  "type": "record",  "namespace": "com.acme.avro",  "fields": [    {"name": "DataSourceInformation",       "type":{        "type":"map",         "values": {          "name":"DataSourceInformation",          "type":"record",           "fields": [            {"name":"DataSourceUuid", "type":["null","string"], "default":null},          &n

I have a cluster of 350+ nodes ( Edge Nodes, Ingestion Nodes, Management Nodes & Worker Nodes) and all these managed by Ambari. All the Management nodes and Worker nodes are up-to-date with the latest Krb5.conf file. However the Edge Nodes and Ingestion Nodes Krb5.conf file is not updated and missing an entry. Hence My question is, If I modify the krb5.conf file in the Edge nodes & Ingestion nodes Manually to match with the same krb5.conf file in the remaining nodes, what would be the impact? Which all components would require restart? How do i go about syncing them up?No master services or client services are installed on these nodes. However they are part of the cluster. Any help would be appreciated.

Hi All,We have created a Hive database and have some external tables on with files placed in S3 bucket. Our Application accesses this data using Impala.Tables are not partitioned and the files are in text/csv format.How should we collect stats using Impala or Hive ?Is it possible to collect stats for Hive tables using Impala and HIve tables stats should only be created to hive. Appreciate your help. Regards.Sandeep Suman

Hi Friends,I have little doubt about Impala using fair schedulre for launching job. I am reading about Impala since 3 months but I never come across about Impala using fair scheduler instead it has its own mechanism for resource allocation. Is there any situation where Impala uses fair scheduler during execution of query?I have one more doubt about IMPALA using YARN. I want to the scenario/condition when we have to use YARN with IMPALA, becuase IMPALA has its own execution engine. When we have to use YARN with IMPALA? I think Llama is meant for that only.

Hello All, Could someone appraise me the significance of Impala Audit & Lineage Logs. We are facing space issue and would like  to know whether we can delete some of those old log files, also the default setting for Impala Daemon Maximum Lineage Log File Size & Impala Daemon Maximum Audit Log File Size is 5000 can this be lowered to 1000 etc. RegardsWert

Hi Friends,I am trying to run IMPALA queries using query options. I am trying to analyze the resultant query attributes. What I have observed that when I run the query for the first time, it took long time than I run the query second time or onwards.I want to know the reason behind difference in time?Does this happen only when table statistics are not available or it happens all the time.Is my observation right? 

Hi,I'm having a little bit of a problem when trying to create a Database and when trying to create a Table.I was following the Hortonworks tutorial for Loading and Querying Data with Hadoop, but when I try to create new Databases or new Tables the Database option show me no results, even if the query to create them was successul.Queries:Database:The Database "TCC" that I created don't appear either.

I'm trying to work with Spark and Hive for HDP 3.0. As I see in some articles, now we have to use Hive Warehouse Connector. Everything works fine except for one proble.I can't write a dataframe directlyh from Spark to Hive using Hive Warehouse connector.I have the following code: val spark = SparkSession      .builder()      .appName("Spark Hive Test Job")      .config("spark.sql.hive.hiveserver2.jdbc.url", "jdbc:hive2://s01.ndtstu.local:2181,m01.ndtstu.local:2181,m02.ndtstu.local:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2;user=hive;password=hive?")      .getOrCreate()                val hive = HiveWarehouseSession.session(spark).build()    hive.createDatabase("testADD", true);    hive.setDatabase("testADD")    hive.createTable("tabletest").ifNotExists()        .column("k", "int")      &nb