After loading csv data into HDFS, I have seen below way to create a Hive external table(textfile format). Followed by creating Impala internal table(parquet format) like the Hive external table. And it works.
My question:
1) Why should one go this roundabout way of creating a Hive table; and then an impala table from it? Why can't we directly create an external impala table (in parquet format)?
2) Is there any issue with sticking to external tables only(without any internal tables) - given that my data is always bulk loaded directly into hdfs?
2) When should one use "from_unixtime(unix_timestamp(as_of_date,"dd-MMM-yy"),'yyyy-MM-dd')"
and store date as string vs storing date as timestamp in Impala?
// sample external table defined below
create EXTERNAL TABLE my_external_table ( Col1 string, as_of_date string, Col3 string ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES ( 'separatorChar' = ',', 'quoteChar' = '"', 'escapeChar' = '\\' ) STORED AS TEXTFILE LOCATION "/data/my_data_files" tblproperties("skip.header.line.count"="1");
// sample internal table defined below
create table my_internal_table like my_external_table stored as parquet;
insert into table my_internal_table select Col1, from_unixtime(unix_timestamp(as_of_date,"dd-MMM-yy"),'yyyy-MM-dd'), Col3 from my_external_table;
... View more
