I would not advise to do this - only if you have no other options and you are sure there are classpath problems (this suggests you had that - likely not having set up the Spark service dependency with Hive). Always check the HS2 logs first what is your problem. With including all the hadoop jars to  SPARK_DIST_CLASSPATH you submit and upload lots of jars to the container classpath unnecessarily, which will slow down the job submission time.  ... View more

When you can't submit Hive on Spark queries, you need to review what is in the HiveServer2 logs. From client end (beeline) it is unfortunately not obvious. In any case you need to make sure that: - Spark service has been enabled as a dependency in Hive service > Configuration - Review Spark related settings in Hive service > Configuration - you have enough resources on the cluster and can submit YARN jobs   Do you have error messages from the HS2 logs?   Thanks  Miklos ... View more

Hi, With CDH 5.x, Hive on Spark2 is still not supported. In CDH 6.x versions only Spark2 is available (there is no more separate parcel for it), so with that you can have Hive on Spark2 and as such it is supported too. ... View more

Hello Harsha,   Based on the stacktrace: 1. HMS fails to start up when it tries to initialize it's token store (" Hive Metastore Delegation Token Store" -> org.apache.hadoop.hive.thrift.ZooKeeperTokenStore). If you do not need this, you may change it to DBTokenStore, but I would go with the next item rather. 2. The "java.lang.NoSuchMethodError" suggests that there is a jar conflict, and multiple versions of a jar exists on the classpath, and org/apache/curator/framework/imps/CreateBuilderImpl is not compiled with the actually loaded org/apache/curator/utils/ThreadUtils class. Please check in which jars do you have these classes   for i in $(find . -name '*.jar') ; do /usr/java/latest/bin/jar tf $i | grep org/apache/curator/framework/imps/CreateBuilderImpl.class && echo $i ; done   for i in $(find . -name '*.jar') ; do /usr/java/latest/bin/jar tf $i | grep org/apache/curator/utils/ThreadUtils.class  && echo $i ; done   and try to find which of these are (mistakenly?) included in the HMS's classpath, use jinfo to get the classpath of HMS:   <java_home_of_hms>/bin/jinfo <pid_of_hms> | grep java.class.path   What Hive/CDH/HDP version are you using? ... View more

Thank you for sharing your solution, I'm glad it is resolved. Yes, it is usually advised to check and go with the latest versions - and this is true for the Teradata connector too. ... View more

That's correct,  Hive Impersonation should be disabled in Hive when using Sentry. Could you achieve the desired results that way? ... View more

Hello Medloh,   At the time of this writing (latest version are CDH 5.13.1 / Spark 2.2.x) Hive on Spark2 is not supported. See our documentation: https://www.cloudera.com/documentation/enterprise/release-notes/topics/cdh_rn_hive_ki.html#hive_on_spark2    "Hive-on-Spark is a CDH component that has a dependency on Spark 1.6. Because CDH components do not have any dependencies on Spark 2, Hive-on-Spark does not work with the Cloudera Distribution of Apache Spark 2."   The referenced Spark 2 announcement was about a Hive issue inside Spark2 (which means it appeared when using HiveContext inside Spark2).   Currently we do not have plans / schedules when Hive on Spark2 will be supported. Until then of course you can use Spark2 (spark2-shell, spark2-submit) and execute Hive queries using HiveContext.   I hope this answers your question.   Best regards  Miklos Szurap Customer Operations Engineer ... View more

Hi,   As far as I see Hive 2.1.1 and Impala 2.7 are still under development (they are not included into any CDH version) so there may be some open integration issues. I've found a reference on field only in HIVE-11553   Regards Miklos Szurap Customer Operations Engineer at Cloudera ... View more

Hi,   Please see: http://www.cloudera.com/documentation/enterprise/5-8-x/topics/admin_hos_oview.html#concept_a5r_vsc_kv   It is normal that your query execution time is slower for the first time, as the Hive query is converted to Spark execution and needs to allocate the YARN containers (Executors in Spark terminology). For the second time the executors are already up and running so it will execute faster. Also the inherent property of Spark is to keep data in memory if possible (so we can call this "caching"), but that depends on multiple factors.   Regards  Miklos Szurap Customer Operations Engineer ... View more

Hi,   Please check your workflow's "job.properties" file, the "nameNode" value should be in format "hdfs://ha-nameservice" (if NameNode HA enabled) or "hdfs://namenode-host:8020" (if there is no NameNode HA). Although your cluster is most probably contains one host only (you use Cloudera quickstart image), the "localhost:8020" would not resolve properly to the NameNode from everywhere. As the error message said, can you also double-check the values "default.fs.name" and "hive.metastore.warehouse.dir" from Cloudera Manager? What are their values?   Regards  Miklos Szurap Customer Operations Engineer at Cloudera ... View more

Hi,   Locking down hive CLI can be done with setting hadoop.proxyuser.hive.groups to a value which restricts hive access to specific groups only - and not to everyone ( * - asterisk for everyone).   http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_sg_hiveserver2_security.html#concept_vxf_pgx_nm   Regards  Miklos Szurap Customer Operations Engineer ... View more

Hi, As far as I see on https://www.cloudera.com/documentation/enterprise/5-7-x/topics/cdh_qs_mrv1_pseudo.html the cloudera-cdh-5-0.x86_64.rpm is related to MRv1. Do you need that? Also do you see the same packages on all the hosts? ... View more

It's not obvious from that output. Let's try from the other end. As you have package based installation can you run the following?   rpm -qa | egrep "cdh|parquet" ... View more

Hi,   You can search for the class with a command like:   for i in `find /usr/lib -name '*.jar'` ; do /usr/java/jdk1.7.0_67-cloudera/bin/jar tf $i | grep "parquet/schema/Type.class" && echo $i ; done You haven't wrote, is it a fresh install? ... View more

Hi,   What version of CDH are you using?   In pre-CDH5.5.4 version there was a known problem,  you may need to set the "ssl_client_ca_certificate" parameter to empty value to disable intra-process SSL/TLS and to enable Impala to start up. See [1] and [2].   Regards  Miklos Szurap Customer Operations Engineer   [1] http://www.cloudera.com/documentation/enterprise/5-5-x/topics/impala_ssl.html [2] https://issues.cloudera.org/browse/IMPALA-2598 ... View more

Hi,   The select * query can run "locally" in a "streaming mode", so no mapreduce job is needed. The select count(*) query on the other hand starts up a mapreduce job, so it involves more nodes. The error message says that the "master1" node cannot connect to "slave1" node because of networking problems.   Please check if the network connectivity from all the hosts to all other hosts are fine (network interface is up and running), you don't use multiple network interfaces and the DNS resultion is working fine.   If you use Cloudera Manager, then please run the "Host inspector" to check these easily.   Regards  Miklos Szurap Customer Operations Engineer ... View more

Hi,   The error message " Could not initialize class com.esri.hadoop.hive.serde.JsonSerde" is not really complete, it does not tell why it could not initialize the class. Maybe it requires some native library?   To get the real error message, either increase the logging of Hive CLI, or try it with beeline - and also please check the logs of Hive Metastore for complete error message.   Regards  Miklos Szurap Customer Operations Engineer ... View more

Hi,   The  NoSuchMethodError suggests that there are two incompatible version of java classes in your classpath (these are usually found in "hive-exec.jar"). Maybe one node in your cluster is misconfigured and contains old hive libraries.   This usually can happen after an upgrade when you migrate from packages to parcel based installation. Please check if the /usr/lib/hive and /usr/lib/hive-hcatalog directories are deleted on the NodeManager hosts (at least rename them first).   Regards  Miklos Szurap Customer Operations Engineer ... View more

Hi,   I have not tried the 5.8 quickstart vm image till now, but if it asks for cluster details, then the options as shown on the screenshot are fine. Use the latest parcel, and if you don't need to experiment with different other connector parcels (you do not need to integrate with those systems) then you can safely leave those at None.   Regards  Miklos Szurap Customer Operations Engineer ... View more

Hi,   Can you double-check that the target table in Teradata DB is defined as decimal(18,2)?   I could reproduce the behavior you see only if the table is defined simply as decimal without scale.     MYUSER=sqoop; MYPSWD=sqoop; MYCONN="jdbc:teradata://tdhost/DATABASE=MYDB"; sqoop list-tables --connect $MYCONN --username $MYUSER --password $MYPSWD sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "create table t1_decimal (col1 varchar(20), col2 decimal)" sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "insert into t1_decimal values ('SAMPLE', 1.234)" sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from t1_decimal"   This returns: ----------------------------------------------- | col1 | col2 | ----------------------------------------------- | SAMPLE | 1 | -----------------------------------------------   But if I define the table as    sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "drop table t1_decimal" sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "create table t1_decimal (col1 varchar(20), col2 decimal(18,2))" sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "insert into t1_decimal values ('SAMPLE', 1.234)" sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from t1_decimal" sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table t1_decimal --target-dir t1_decimal --delete-target-dir -m 1 --split-by col1 sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "delete from t1_decimal" hdfs dfs -text t1_decimal/* This returns: SAMPLE,1.23   If I export it to Teradata DB table sqoop export --connect $MYCONN --username $MYUSER --password $MYPSWD --table t1_decimal --export-dir t1_decimal -m 1 sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from t1_decimal" It returns the expected, same value ----------------------------------------------- | col1 | col2 | ----------------------------------------------- | SAMPLE | 1.23 | ----------------------------------------------- It maintans the precision with bigger numbers, like 12.2345 -> will be exported like 12.23   My environment is CDH 5.5, TD 14.00    Miklos Szurap Customer Operations Engineer ... View more