Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Solved
Go to solution
Hive partition getting dropped while HDFS service is down
Labels:
- Labels:
-
Apache Hive
Contributor
Created on ‎11-20-2018 03:47 AM - edited ‎09-16-2022 06:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
i'm not sure what's going wrong out here ,ideally it should not happen
but , when i happen to execute the below query when HDFS service is down , i would notice the partition being dropped despite of below error
Query: alter table fenet5_dev.dw_malicious_events drop partition (occurred_month = 201808) purge ERROR: ImpalaRuntimeException: Error making 'dropPartition' RPC to Hive Metastore: CAUSED BY: MetaException: Got exception: java.net.ConnectException Call From hpc143 to hpc123:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
1 ACCEPTED SOLUTION
Guru
Created ‎11-20-2018 07:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I think it is as it is. The drop of the partition is an operation on the Hive Metastore (remove the records from the backend database) and then it tries to contact the NameNode to remove the directory.
Unfortunately this is not an atomic operation, so it will not "roll back" whent he NN is not accessible.
Unfortunately this is not an atomic operation, so it will not "roll back" whent he NN is not accessible.
2 REPLIES 2
Guru
Created ‎11-20-2018 07:54 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I think it is as it is. The drop of the partition is an operation on the Hive Metastore (remove the records from the backend database) and then it tries to contact the NameNode to remove the directory.
Unfortunately this is not an atomic operation, so it will not "roll back" whent he NN is not accessible.
Unfortunately this is not an atomic operation, so it will not "roll back" whent he NN is not accessible.
Contributor
Created ‎11-21-2018 10:52 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Tomas79, Thank you for the inputs
