Spark cannot read hive orc table
Hello all,
I cannot read data from hive orc table and load to dataframe. If someone know, could you help me to fix it? Below is my scripts:
from pyspark import SparkContext, SparkConf
from pyspark.conf import SparkConf
from pyspark.sql import SparkSession
from pyspark.sql import HiveContext,SQLContext
spark = SparkSession.builder.appName("Testing....").enableHiveSupport().getOrCreate()
hive_context = HiveContext(spark)
sqlContext = SQLContext(spark)
df_pgw=hive_context.sql("select * from orc_table")
Hive Session ID = 79c9e6c0-1649-41dc-9aea-493c0f62d046
22/07/20 11:50:52 WARN SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
22/07/20 11:50:56 WARN HiveMetastoreCatalog: Unable to infer schema for table orc_table from file format ORC (inference mode: INFER_AND_SAVE). Using metastore schema.
df_pgw.show()
=> ....Don't have data presents
Thanks,
