Reply
Highlighted
Explorer
Posts: 19
Registered: ‎04-23-2019

how to access hive database/tables through spark and pyspark shell?

 
Posts: 45
Topics: 0
Kudos: 13
Solutions: 4
Registered: ‎07-30-2018

Re: how to access hive database/tables through spark and pyspark shell?

Hi Harish,

You can create a hive context and can access the hive table.

Example Program:

from pyspark.sql import HiveContext
hive_context = HiveContext(sc)
sample = hive_context.table("default.<tablename>")
sample.show()

Reference Link: https://stackoverflow.com/questions/36051091/query-hive-table-in-pyspark
Explorer
Posts: 8
Registered: ‎02-13-2018

Re: how to access hive database/tables through spark and pyspark shell?

from spark or pyspark shell use the below commands to access hive database objects.

 

spark.sql("show databases;")

spark.sql("select * from databasename.tablename;")

or

spark.read.table("databasename.tablename")

 

You can give any query inside spark.sql which will give you results.