Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive partition getting dropped while HDFS service is down

SOLVED Go to solution

Hive partition getting dropped while HDFS service is down

Explorer

i'm not sure what's going wrong out here ,ideally it should not happen

but , when i happen to execute the below query when HDFS service is down , i would notice the partition being dropped despite of below error

 

Query: alter table fenet5_dev.dw_malicious_events drop partition (occurred_month = 201808) purge
ERROR: ImpalaRuntimeException: Error making 'dropPartition' RPC to Hive Metastore:
CAUSED BY: MetaException: Got exception: java.net.ConnectException Call From hpc143 to hpc123:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Hive partition getting dropped while HDFS service is down

Master Collaborator
I think it is as it is. The drop of the partition is an operation on the Hive Metastore (remove the records from the backend database) and then it tries to contact the NameNode to remove the directory.
Unfortunately this is not an atomic operation, so it will not "roll back" whent he NN is not accessible.
2 REPLIES 2

Re: Hive partition getting dropped while HDFS service is down

Master Collaborator
I think it is as it is. The drop of the partition is an operation on the Hive Metastore (remove the records from the backend database) and then it tries to contact the NameNode to remove the directory.
Unfortunately this is not an atomic operation, so it will not "roll back" whent he NN is not accessible.

Re: Hive partition getting dropped while HDFS service is down

Explorer
@Tomas79, Thank you for the inputs