Hi everyone, I am getting crazy trying to figure out, why I cannot read a Hive External Table, which points to a directory with parquet files. The parquet files are created with a Spark program like this: eexTable.repartition(1).write.mode("append").save(dataPath.concat(eexFileName)) I created an external table using this dll: CREATE EXTERNAL TABLE my_db.eex_actual_plant_gen_line (
meta_date timestamp,
meta_directory string ,
meta_filename string,
meta_host string,
meta_numberofrecords int,
meta_recordnumber int,
meta_sequenceid string,
country string,
source string,
data_timestamp timestamp,
actual_generation double,
publication_timestamp timestamp,
modification_timestamp timestamp,
created_on timestamp
)
COMMENT 'External table eex transparency actual plant generation line'
ROW FORMAT SERDE
'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
LOCATION '/development/data/EEX/transparency/production/usage' I am able to query this table using Ambari or CLI but When I try to use spark I can retrieve the table schema but no rows are returned: import org.apache.spark.sql.{ Row, SaveMode, SparkSession }
import org.apache.spark.sql.functions._
val warehouseLocation = "/apps/hive/warehouse"
val spark = SparkSession
.builder()
.appName("EEX_Trans")
.config("spark.sql.warehouse.dir", warehouseLocation)
.config("hive.metastore.uris", "thrift://myserver1:9083,thrift://myserver2:9083")
.enableHiveSupport()
.getOrCreate()
val hadoopConf = spark.sparkContext.hadoopConfiguration
hadoopConf.set("mapreduce.input.fileinputformat.input.dir.recursive", "true")
spark.sql("Select * from dev_sdsp.facilities").count() I cannot find the error and I already read 1000 posts without luck. Any comment will be appreciated. Kind regards, Paul
... View more
