@AndreaCavenago For that you will have to check if the connection is getting interrupted/closed between the client and hiveserver2. Without thorough log analysis, it will be difficult to answer that. Could you open a support case for the same? ... View more

@itdm_bmi We are still seeing the error "java.lang.ClassNotFoundException: Class org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe" This means, you have not uploaded the "hive-contrib" jar to Hive classpath as mentioned here. If you are on CDP(>7.1.5 onward) then, we will not have to upload hive-contrib jar to the class path. This serde class is already added to Hive native serde list. You just need to alter the original table, and change the serde(Note: only applies to CDP version > 7.1.5) ALTER TABLE <table name> SET SERDE 'org.apache.hadoop.hive.serde2.MultiDelimitSerDe' If you are on an older version, do consider uploading to hive-contrib jar to hive classpath by uploading this to aux jar path location. ... View more

@AndreaCavenago Does this error appear every time we run this spark-submit command? As this is a warning message, and it does not have any real impact, we can avoid it by changing the log level. In the script.py file, add the following two lines: from pyspark import SparkContext SparkContext.setLogLevel("ERROR") This will avoid the WARN message. But it will still be good to address the actual issue. ... View more

@novice_tester when you create a database, it takes the external warehouse directory path from hive.metastore.warehouse.external.dir, and sets that as the database LOCATION for external tables only(when created without setting location clause). Please note, there is another MANAGEDLOCATION field that is blank.   e.g. describe database extended question; +-----------+----------+----------------------------------------------------+------------------+-------------+-------------+-------------+ | db_name | comment | location | managedlocation | owner_name | owner_type | parameters | +-----------+----------+----------------------------------------------------+------------------+-------------+-------------+-------------+ | question | | hdfs://node2.cloudera.com:8020/warehouse/tablespace/external/hive/question.db | | hive | USER | | +-----------+----------+----------------------------------------------------+------------------+-------------+-------------+-------------+  A Managedlocation can be set on a database if you want to use a different location for managed tables other than hive.metastore.warehouse.dir. Otherwise it is kept blank. I hope this helps.   To summarize, you do not need to worry about the LOCATION of a database. It can host both managed and external tables. When you create a managed table under the same DB, it will pick the path set in hive.metastore.warehouse.dir, irrespective of the database Location, and when you create an external table without the location clause, it gets stored in the DB LOCATION path. ... View more

@vaibhavgokhale You could try: --conf spark.sql.hive.conf.list="tez.queue.name=queue1"   ... View more

@Choolake Try this: count1=$(beeline -u "jdbc:hive2://dev-lisa.realm.com:10000/default;principal=hive/dev-lisa.intranet.slt.com.lk@REALM.COM;ssl=true;sslTrustStore=/var/lib/cloudera-scm-agent/agent-cert/cm-auto-global_truststore.jks" --showHeader=false --silent=true --outputformat=tsv2 -e 'SELECT count(*) from table_name;')  These beeline flags will remove all the unnecessary texts from the stdout. Compute count2 the same way. ... View more