Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hive context on spark in quickstart vm

Highlighted

hive context on spark in quickstart vm

Explorer

I am unable to initialize hive context in spark. Hive runs as standalone

 

I changed the hive-site.xml in spark/conf as:

  <property>    <name>hive.metastore.warehouse.dir</name>
    <value>/user/hive/warehouse</value>
    <description>    Local or HDFS directory where Hive keeps table contents.
    </description>  </property>
  <property>   <name>javax.jdo.PersistenceManagerFactoryClass</name>
   <value>org.datanucleus.api.jdo.JDOPersistenceManagerFactory</value>
   <description>class implementing the jdo persistence</description>  </property>

  <!--<property>   <name>hive.enable.spark.execution.engine</name>
   <value>true</value>  </property> -->

  <!-- modded   -->
  <property>    <name>javax.jdo.option.ConnectionURL</name>
    <!--<value>jdbc:mysql://127.0.0.1/metastore?createDatabaseIfNotExist=true</value>-->
   <value>jdbc:derby:;databaseName=/home/cloudera/metastore_db;create=true</value>
    <description>JDBC connect string for a JDBC metastore</description>  </property>

    <!--  modded   -->
  <property>    <name>javax.jdo.option.ConnectionDriverName</name>
    <!--<value>com.mysql.jdbc.Driver</value>-->
   <value>org.apache.derby.jdbc.EmbeddedDriver</value>
    <description>Driver class name for a JDBC metastore</description>  </property>

 <!-- <property>    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>  </property>
  <property>    <name>javax.jdo.option.ConnectionPassword</name>
    <value>cloudera</value>  </property>  -->

<property> <name>hive.hwi.war.file</name> <value>/usr/lib/hive/lib/hive-hwi-0.8.1-cdh4.0.0.jar</value> <description>This is the WAR file with the jsp content for Hive Web Interface</description> </property> <property> <name>datanucleus.fixedDatastore</name> <value>true</value> </property> <property> <name>datanucleus.autoCreateSchema</name> <value>false</value> </property> <property> <name>hive.metastore.schema.verification</name> <value>true</value> </property> <property> <name>hive.metastore.uris</name> <value>thrift://127.0.0.1:9083</value> <description>IP address (or fully-qualified domain name) and port of the metastore host</description> </property> <!-- added by Deepak --> <property> <name>dfs.client.read.shortcircuit</name> <value>true</value> </property> </configuration>

In spark-shell:

import org.apache.spark.sql.hive.HiveContext
val hCtx= new org.apache.spark.sql.hive.HiveContext(sc)
val rCH= hCtx.hql("select * from mytable")

Errors:

17/08/24 17:29:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Spark context available as sc (master = local[*], app id = local-1503620964053).
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] java.io.FileNotFoundException: derby.log (Permission denied)
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.replication.master in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.resultSetStatisticsFactory in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.NoneAuthentication in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.lockManagerJ6 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.rawStore.log.readonly in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.jdbcJ8 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.resourceAdapterJ2 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.validation in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.jdbcJ6 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.lcf in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.optimizer in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.XPLAINFactory in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.javaCompiler in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.tcf in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.dataDictionary in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.jdk.cacheManagerJ6 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.ef in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.dvfJ2 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.jdk.dvfJ2 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.daemon in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.cacheManagerJ6 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.rawStore.data.genericJ4 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.timerJ1 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.jdk.rawStore.data.genericJ4 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.rawStore.data.genericJ1 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.basicAuthentication in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.jdk.rawStore.data.genericJ1 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.classes.JNDIAuthentication in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.database in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.rawStore.transaction in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.streams in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.classes.dvfJ2 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.jdk.mgmt.jmx in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.nativeAuthentication in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.jdk.cryptographyJ2 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.access.sort in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.classes.jdbcJ8 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.classes.cryptographyJ2 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.classes.jdbcJ6 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.classes.rawStore.data.genericJ4 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.jdk.jdbcJ8 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.jdk.lockManagerJ6 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.cryptographyJ2 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.jdk.jdbcJ6 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.specificAuthentication in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.mgmt.jmx in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.netServer.autoStart in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.access.uniquewithduplicatenullssort in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.classManagerJ2 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.JNDIAuthentication in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.rawStore.log in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.access.heap in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.database.slave in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.jdk.resourceAdapterJ2 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.mgmt.null in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.env.classes.resourceAdapterJ2 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.access in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.replication.slave in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.lf in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.uuidJ1 in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.access.btree in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] Ignored duplicate property derby.module.rawStore in jar:file:/usr/lib/flume-ng/lib/derby-10.11.1.1.jar!/org/apache/derby/modules.properties
Thu Aug 24 17:29:28 PDT 2017 Thread[main,5,main] 
----------------------------------------------------------------
Thu Aug 24 17:29:28 PDT 2017:
Booting Derby version The Apache Software Foundation - Apache Derby - 10.11.1.1 - (1616546): instance a816c00e-015e-16cb-2ebd-00001f7c1fd0 
on database directory /tmp/spark-e1679bcc-90a7-4a58-b498-1b93233e5694/metastore with class loader sun.misc.Launcher$AppClassLoader@7d487b8b 
Loaded from file:/usr/lib/hive/lib/derby-10.11.1.1.jar
java.vendor=Oracle Corporation
java.runtime.version=1.7.0_67-b01
user.dir=/
os.name=Linux
os.arch=amd64
os.version=2.6.32-573.el6.x86_64
derby.system.home=null
Database Class Loader started - derby.database.classpath=''
17/08/24 17:29:37 WARN metadata.Hive: Failed to register all functions.
java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1530)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3230)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3249)
	at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3474)
	at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:225)
	at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:209)
	at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:332)
	at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:293)
	at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:268)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:529)
	at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:204)
	at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
	at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:220)
	at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:210)
	at org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:465)
	at org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:464)
	at org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)
	at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:342)
	at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
	at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1033)
	at $line4.$read$$iwC$$iwC.<init>(<console>:15)
	at $line4.$read$$iwC.<init>(<console>:24)
	at $line4.$read.<init>(<console>:26)
	at $line4.$read$.<init>(<console>:30)
	at $line4.$read$.<clinit>(<console>)
	at $line4.$eval$.<init>(<console>:7)
	at $line4.$eval$.<clinit>(<console>)
	at $line4.$eval.$print(<console>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1045)
	at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1326)
	at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:821)
	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:852)
	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:800)
	at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
	at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
	at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
	at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:133)
	at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
	at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:305)
	at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
	at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
	at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:160)
	at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
	at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
	at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)
	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
	at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
	at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1064)
	at org.apache.spark.repl.Main$.main(Main.scala:35)
	at org.apache.spark.repl.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:730)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1528)
	... 73 more
Caused by: MetaException(message:Version information not found in metastore. )
	at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7160)
	at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7141)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
	at com.sun.proxy.$Proxy22.verifySchema(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:627)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:675)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:484)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5995)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:203)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
	... 78 more
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:556)
	at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:204)
	at org.apache.spark.sql.hive.client.IsolatedClientLoader.createClient(IsolatedClientLoader.scala:238)
	at org.apache.spark.sql.hive.HiveContext.executionHive$lzycompute(HiveContext.scala:220)
	at org.apache.spark.sql.hive.HiveContext.executionHive(HiveContext.scala:210)
	at org.apache.spark.sql.hive.HiveContext.functionRegistry$lzycompute(HiveContext.scala:465)
	at org.apache.spark.sql.hive.HiveContext.functionRegistry(HiveContext.scala:464)
	at org.apache.spark.sql.UDFRegistration.<init>(UDFRegistration.scala:40)
	at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:342)
	at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:90)
	at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1033)
	at $iwC$$iwC.<init>(<console>:15)
	at $iwC.<init>(<console>:24)
	at <init>(<console>:26)
	at .<init>(<console>:30)
	at .<clinit>(<console>)
	at .<init>(<console>:7)
	at .<clinit>(<console>)
	at $print(<console>)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1045)
	at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1326)
	at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:821)
	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:852)
	at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:800)
	at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
	at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
	at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
	at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:133)
	at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
	at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:305)
	at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
	at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
	at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:160)
	at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
	at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
	at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991)
	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
	at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945)
	at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
	at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945)
	at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1064)
	at org.apache.spark.repl.Main$.main(Main.scala:35)
	at org.apache.spark.repl.Main.main(Main.scala)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:730)
	at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
	at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:214)
	at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:332)
	at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:293)
	at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:268)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:529)
	... 62 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1530)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:67)
	at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:82)
	at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3230)
	at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3249)
	at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3474)
	at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:225)
	at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:209)
	... 66 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
	at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1528)
	... 73 more
Caused by: MetaException(message:Version information not found in metastore. )
	at org.apache.hadoop.hive.metastore.ObjectStore.checkSchema(ObjectStore.java:7160)
	at org.apache.hadoop.hive.metastore.ObjectStore.verifySchema(ObjectStore.java:7141)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:606)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:114)
	at com.sun.proxy.$Proxy22.verifySchema(Unknown Source)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:627)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:675)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:484)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:78)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5995)
	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:203)
	at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:74)
	... 78 more
<console>:16: error: not found: value sqlContext
         import sqlContext.implicits._
                ^
<console>:16: error: not found: value sqlContext
         import sqlContext.sql
                ^
scala> 

Can somebody help how to start hive context and load hive tables?

 

Appreciate your help in advance.