Reply
Highlighted
Contributor
Posts: 40
Registered: ‎03-19-2018
Accepted Solution

Hive partition getting dropped while HDFS service is down

i'm not sure what's going wrong out here ,ideally it should not happen

but , when i happen to execute the below query when HDFS service is down , i would notice the partition being dropped despite of below error

 

Query: alter table fenet5_dev.dw_malicious_events drop partition (occurred_month = 201808) purge
ERROR: ImpalaRuntimeException: Error making 'dropPartition' RPC to Hive Metastore:
CAUSED BY: MetaException: Got exception: java.net.ConnectException Call From hpc143 to hpc123:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
Master
Posts: 377
Registered: ‎07-01-2015

Re: Hive partition getting dropped while HDFS service is down

I think it is as it is. The drop of the partition is an operation on the Hive Metastore (remove the records from the backend database) and then it tries to contact the NameNode to remove the directory.
Unfortunately this is not an atomic operation, so it will not "roll back" whent he NN is not accessible.
Contributor
Posts: 40
Registered: ‎03-19-2018

Re: Hive partition getting dropped while HDFS service is down

@Tomas79, Thank you for the inputs
Announcements