Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

create hive external incremental update table with partitions

create hive external incremental update table with partitions

Hi,

I want to create hive external table with multiple partitioning collumns and should allow dynamic partitions. Suppose i create a table

set hive.exec.dynamic.partition=true

set hive.exec.dynamic.partition.mode=nonstrict

create external table sample1( name string,age int) partitioned by (year int, month int,day int) location '/path to hdfs storage';

now i just run

msck repair table sample1;

So, is it correct method to achieve it and if its not can you help me to do it.

Warm regards

Sidharth

2 REPLIES 2
Highlighted

Re: create hive external incremental update table with partitions

Super Guru
@Sidharth Kumar

MSCK(Metastore check command) it is used when the partitions are added directly to HDFS directory(say by using hadoop fs -put command), To make aware of Hive metastore about these newly added partitions we are using msck repair table command.

msck repair table <table-name>;

(or)

We can add partitions by alter table as follows but in this case we need to run the command on each of the newly added partitions.

alter table partition_table add partition(<partition-name>) location '<directory-path>';

These commands won't create partitions or load data to the partitions,By running these commands we are going to add partitions to the table update the metastore with newly added partitions.

If you want to create and load the data to partitions refer the following link.

https://community.hortonworks.com/questions/145863/hive-partitions-based-on-date-from-timestamp.html...

Highlighted

Re: create hive external incremental update table with partitions

@Shu I know this is an old question but I have a similar use case and wanted to follow up to your response. If I run alter table add partition command, it will create an exclusive lock on that partition not allowing the end user to query that partition during loading. If I run MSCK command, will it create an exclusive lock or is the user prone to a dirty read while the command is running?

Don't have an account?
Coming from Hortonworks? Activate your account here