Support Questions

Find answers, ask questions, and share your expertise

Hive partition getting dropped while HDFS service is down

avatar
Contributor

i'm not sure what's going wrong out here ,ideally it should not happen

but , when i happen to execute the below query when HDFS service is down , i would notice the partition being dropped despite of below error

 

Query: alter table fenet5_dev.dw_malicious_events drop partition (occurred_month = 201808) purge
ERROR: ImpalaRuntimeException: Error making 'dropPartition' RPC to Hive Metastore:
CAUSED BY: MetaException: Got exception: java.net.ConnectException Call From hpc143 to hpc123:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
1 ACCEPTED SOLUTION

avatar
I think it is as it is. The drop of the partition is an operation on the Hive Metastore (remove the records from the backend database) and then it tries to contact the NameNode to remove the directory.
Unfortunately this is not an atomic operation, so it will not "roll back" whent he NN is not accessible.

View solution in original post

2 REPLIES 2

avatar
I think it is as it is. The drop of the partition is an operation on the Hive Metastore (remove the records from the backend database) and then it tries to contact the NameNode to remove the directory.
Unfortunately this is not an atomic operation, so it will not "roll back" whent he NN is not accessible.

avatar
Contributor
@Tomas79, Thank you for the inputs