Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Impala insert overwrite corruption possible

avatar
New Contributor

The questions is actually simple:
What happens if during an INSERT OVERWRITE of a table/partition an error occurs: power failure , underlying files is removed and so on. Is the table/partition corrupt or is it automatically restored because the metadata of the impala table is not up te date on the nodes.
Situation
We have a partitioned table ( 144 partitions a day ) and each partition has about 700-800 files.
We do an automatic INSERT OVERWRITE of partitions that are not to be changed anymore. The number of files reduces to 4-5, very nice.

1 REPLY 1

avatar
Moderator

Hello @dirkj ,

 

the answer to your question is in the source code of the INSERT OVERWRITE in a form of comments under FinalizeHdfsInsert.

 

Please let us know if you need further clarification on it!

 

Best regards:

Ferenc


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community: