@Geoffrey Shelton Okot No luck. Pre-emption is already enabled via yarn config and all other prerequisite has completed. Hive interactive query service is running fine. Still 19/01/03 05:16:45 INFO RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=vinay@TEST.COM (auth:KERBEROS) retries=1 delay=5 lifetime=0 19/01/03 05:16:47 INFO CodeGenerator: Code generated in 294.781928 ms 19/01/03 05:16:47 INFO CodeGenerator: Code generated in 18.011739 ms +------------+ |databaseName| + ------------+ | default| +------------+ ... View more

@Geoffrey Shelton Okot Ohh. I did not enable the pre-emption via yarn config, It is only point which is pending. Rest of part, i have completed. let me check with enable yarn pre-emption. Will update you once done it. ... View more

Hi Subhash below is code from pyspark import SparkConf from pyspark.sql import SparkSession, HiveContext from pyspark.sql import functions as fn from pyspark.sql.functions import rank,sum,col from pyspark.sql import Window sparkSession = (SparkSession .builder .master("local") .appName('sprk-job') .enableHiveSupport() .getOrCreate()) sparkSession.sql("show databases").show() sparkSession.stop() Even i'm also trying from spark-shell. ... View more

Hi Subhash I have already added spark user for access the all database by ranger and all HDFS storage path. ... View more

@Geoffrey Shelton Okot Could you please confirm do we really need to enable Interactive query? because after enable Interactive query, i'm unable to start interactive query service. Below are the logs: 2019-01-02T08:36:41,455 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2019-01-02T08:36:43,462 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2019-01-02T08:36:45,469 WARN [main] cli.LlapStatusServiceDriver: Watch mode enabled and got YARN error. Retrying.. 2019-01-02T08:36:47,476 INFO [main] LlapStatusServiceDriverConsole: LLAP status unknown ... View more

@Geoffrey Shelton Okot Hive and spark client has already installed on hive and spark node. ... View more

Hi I'm facing same issue. I have also added following properties in spark configuration. But still it's not working. spark.sql.hive.hiveserver2.jdbc.url:jdbc:hive2://hive_server_FQDN:10000/ spark.datasource.hive.warehouse.metastoreUri: thrift://hive_server_FQDN:9083 spark.datasource.hive.warehouse.load.staging.dir: /tmp spark.hadoop.hive.llap.daemon.service.hosts: @llap0 spark.hadoop.hive.zookeeper.quorum: hadoop3.test.com:2181 spark.sql.hive.hiveserver2.jdbc.url.principal: hive/_HOST@TEST.COM ... View more

@Geoffrey Shelton Okot I had copied manually /etc/hive/conf/hive-site.xml to /etc/spark2/conf/ and restarted spark service. After restart /etc/spark2/conf/hive-site.xml changed to previous hive-site.xml which i had replaced. Latest status is, still not able to see hive database by spark. even i have also added below properties in spark configuration: spark.sql.hive.hiveserver2.jdbc.url.principal spark.hadoop.hive.zookeeper.quorum spark.hadoop.hive.llap.daemon.service.hosts spark.datasource.hive.warehouse.load.staging.dir spark.datasource.hive.warehouse.metastoreUri spark.sql.hive.hiveserver2.jdbc.url ... View more

@subhash parise i have also tried spark-shell --conf spark.sql.hive.hiveserver2.jdbc.url="jdbc:hive2://************************:10000/" spark.datasource.hive.warehouse.load.staging.dir="/tmp" spark.hadoop.hive.zookeeper.quorum="ip.************:2181" --jars /usr/hdp/current/hive_warehouse_connector/hive-warehouse-connector-assembly-1.0.0.3.0.0.0-1634.jar but same result. Only Default database is visible. ... View more

@Geoffrey Shelton Okot I had already defined spark.sql.hive.hiveserver2.jdbc.url.principal=hive/_HOST@TEST.COM in configuration. ... View more