Impalad daemon 2.3 crashing regularly

[#1]

Hi,

We have had several daemon restarts and the last log entry before the restart was:

I1125 02:07:05.337430 32232 status.cc:114] File hdfs://nameservice1/bigdata/parquet/cmdb/cmdb_node_stats_1448445780889.parquet.snappy is invalid. Missing metadata.
@ 0x78b793 (unknown)
@ 0xbede4b (unknown)
@ 0xbee501 (unknown)
@ 0xbbb5aa (unknown)
@ 0xaa4faa (unknown)
@ 0xaa7130 (unknown)
@ 0xca79b3 (unknown)
@ 0x7f5ef0d3e9d1 start_thread
@ 0x7f5eefc9c9dd clone

Would this error be responsible for the restart? (Not too obvious looking at the code)

Would a refresh table cmd executed while the parquet table was being written and be incomplete cause this restart?

All indications points to this but would like a second opinion :)

Thanks in advance!

Stephane

[#2]

Hi Stefanie,

Ideally, you don't want refresh statements to be interleaved with inserts. The error message indicates some stale metadata but we can't tell for sure the root cause of it. I would recommend using refresh statements as serialization points and see if the problem persists. 

Regards,

Dimitris