We have had several daemon restarts and the last log entry before the restart was:
I1125 02:07:05.337430 32232 status.cc:114] File hdfs://nameservice1/bigdata/parquet/cmdb/cmdb_node_stats_1448445780889.parquet.snappy is invalid. Missing metadata.
@ 0x78b793 (unknown)
@ 0xbede4b (unknown)
@ 0xbee501 (unknown)
@ 0xbbb5aa (unknown)
@ 0xaa4faa (unknown)
@ 0xaa7130 (unknown)
@ 0xca79b3 (unknown)
@ 0x7f5ef0d3e9d1 start_thread
@ 0x7f5eefc9c9dd clone
Would this error be responsible for the restart? (Not too obvious looking at the code)
Would a refresh table cmd executed while the parquet table was being written and be incomplete cause this restart?
All indications points to this but would like a second opinion :)
Thanks in advance!
Ideally, you don't want refresh statements to be interleaved with inserts. The error message indicates some stale metadata but we can't tell for sure the root cause of it. I would recommend using refresh statements as serialization points and see if the problem persists.