Support Questions
Find answers, ask questions, and share your expertise

Impala insert overwrite corruption possible

New Contributor

The questions is actually simple:
What happens if during an INSERT OVERWRITE of a table/partition an error occurs: power failure , underlying files is removed and so on. Is the table/partition corrupt or is it automatically restored because the metadata of the impala table is not up te date on the nodes.
Situation
We have a partitioned table ( 144 partitions a day ) and each partition has about 700-800 files.
We do an automatic INSERT OVERWRITE of partitions that are not to be changed anymore. The number of files reduces to 4-5, very nice.

1 REPLY 1

Moderator

Hello @dirkj ,

 

the answer to your question is in the source code of the INSERT OVERWRITE in a form of comments under FinalizeHdfsInsert.

 

Please let us know if you need further clarification on it!

 

Best regards:

Ferenc


Ferenc Erdelyi, Technical Solutions Manager

Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.

Learn more about the Cloudera Community:

; ;