I am trying to access the already existing table in hive by using pyspark
e.g. in hive table is existing name as "department" in default database.
err msg :-
18/10/15 22:01:23 WARN shortcircuit.DomainSocketFactory: The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
18/10/15 22:02:35 WARN metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.1.0-cdh5.13.0
18/10/15 22:02:38 WARN metastore.ObjectStore: Failed to get database default, returning NoSuchObjectException
I checked the below files, they are same.
Any help on how to set up the HiveContext from pyspark is highly appreciated.
Just in case someone still needs the solution, here is what i tried and it works.
spark-shell --driver-java-options "-Dhive.metastore.uris=thrift://quickstart:9083"
I am using spark 1.6 with cloudera vm.
val df=sqlContext.sql("show databases")
You should be able to see all the databases in hive. I hope it helps.