The questions is actually simple: What happens if during an INSERT OVERWRITE of a table/partition an error occurs: power failure , underlying files is removed and so on. Is the table/partition corrupt or is it automatically restored because the metadata of the impala table is not up te date on the nodes. Situation We have a partitioned table ( 144 partitions a day ) and each partition has about 700-800 files. We do an automatic INSERT OVERWRITE of partitions that are not to be changed anymore. The number of files reduces to 4-5, very nice.
... View more