question Re: Hive partition getting dropped while HDFS service is down in Archives of Support Questions (Read Only)

Hive partition getting dropped while HDFS service is down

Johnny_Bach — Fri, 16 Sep 2022 13:54:51 GMT

i'm not sure what's going wrong out here ,ideally it should not happen

but , when i happen to execute the below query when HDFS service is down , i would notice the partition being dropped despite of below error

Query: alter table fenet5_dev.dw_malicious_events drop partition (occurred_month = 201808) purge
ERROR: ImpalaRuntimeException: Error making 'dropPartition' RPC to Hive Metastore:
CAUSED BY: MetaException: Got exception: java.net.ConnectException Call From hpc143 to hpc123:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused

Re: Hive partition getting dropped while HDFS service is down

Tomas79 — Tue, 20 Nov 2018 15:54:45 GMT

I think it is as it is. The drop of the partition is an operation on the Hive Metastore (remove the records from the backend database) and then it tries to contact the NameNode to remove the directory.
Unfortunately this is not an atomic operation, so it will not "roll back" whent he NN is not accessible.

Re: Hive partition getting dropped while HDFS service is down

Johnny_Bach — Thu, 22 Nov 2018 06:52:25 GMT

@Tomas79, Thank you for the inputs