We are one specific query, "insert overwrite select * from a partitions" that usually take < 2min. Occasionaly , it takes over 5 min for unknown reason. Looking at the query profile, we see the following below.  Throughput of < then 600kb sec is unexpected since we have 10GB network... So what could explain this?  Is it releated to  ThriftTransmitTime? Thanks   DataStreamSender (dst_id=1) (5.0m) AsyncTotalTime: 0ns BytesSent: 169.9 MiB InactiveTotalTime: 0ns NetworkThroughput(*): 595.0 KiB/s OverallThroughput: 585.7 KiB/s PeakMemoryUsage: 72.0 KiB SerializeBatchTime: 2.77s ThriftTransmitTime(*): 4.9m TotalTime: 5.0m UncompressedRowBatchSize: 547.1 MiB ... View more

The following query is getting hdfs error: insert overwrite bigdata.nodestats_swap partition (year,month,day,hour) select * from bigdata.nodestats where year=2016 and month=4 and day=5 and hour=11   It does not happen very often but would like to understand why\what happens. Below are the imala and namenode logs.  Thanks in advance.   Impala log: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hive/warehouse/bigdata.db/ nodestats _swap/_impala_insert_staging/724a04f162c946f2_a85c6f0bcd88e99f/.724a04f162c946f2-a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/724a04f162c946f2-a85c6f0bcd88e9a9_1910094467_data.3.parq (inode 249154603): File does not exist. Holder DFSClient_NONMAPREDUCE_-569971520_1 does not have any open files.         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3602)         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3399)         at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3255)         at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:667)         at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:212)         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:483)         at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)         at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)         at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)         at java.security.AccessController.doPrivileged(Native Method)         at javax.security.auth.Subject.doAs(Subject.java:415)         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)           at org.apache.hadoop.ipc.Client.call(Client.java:1468)         at org.apache.hadoop.ipc.Client.call(Client.java:1399)         at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)         at com.sun.proxy.$Proxy14.addBlock(Unknown Source)         at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399)         at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)         at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)         at com.sun.proxy.$Proxy15.addBlock(Unknown Source)         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544)         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600) I0405 06:24:37.250735 19067 runtime-state.cc:230] Error from query 724a04f162c946f2:a85c6f0bcd88e99f: Failed to close HDFS file: hdfs://nameservice1/user/hive/warehouse/bigdata.db/ nodestats _swap/_impala_insert_staging/724a04f162c946f2_a85c6f0bcd88e99f//.724a04f162c946f2-a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/724a04f162c946f2-a85c6f0bcd88e9a9_1910094467_data.3.parq Error(255): Unknown error 255     Relevant Namenode logs: 2016-04-05 06:23:39,449 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/ 724a04f162c946f2 _a85c6f0bcd88e99f/. 724a04f162c946f2 -a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/ 724a04f162c946f2 -a85c6f0bcd88e9a9_1910094467_data.2.parq. BP-1123871397-10.216.32.10-1433806145781 blk_1190477181_116736497{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-caeb90ae-cc3d-4142-9ac1-d83290ba9d4a:NORMAL:10.216.32.105:50010|RBW], ReplicaUnderConstruction[[DISK]DS-2a57f3a9-b316-481f-a3a8-d2430f07dba5:NORMAL:10.216.34.101:50010|RBW], ReplicaUnderConstruction[[DISK]DS-f90cdf95-8c43-4e81-ad37-d2c0dc384634:NORMAL:10.216.34.105:50010|RBW]]}   2016-04-05 06:23:40,770 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/ 724a04f162c946f2 _a85c6f0bcd88e99f/. 724a04f162c946f2 -a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/ 724a04f162c946f2 -a85c6f0bcd88e9a9_1910094467_data.2.parq is closed by DFSClient_NONMAPREDUCE_-569971520_1   2016-04-05 06:24:37,163 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:impala (auth:SIMPLE) cause:org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/ 724a04f162c946f2 _a85c6f0bcd88e99f/. 724a04f162c946f2 -a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/ 724a04f162c946f2 -a85c6f0bcd88e9a9_1910094467_data.3.parq (inode 249154603): File does not exist. Holder DFSClient_NONMAPREDUCE_-569971520_1 does not have any open files. 2016-04-05 06:24:37,163 INFO org.apache.hadoop.ipc.Server: IPC Server handler 249 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 10.216.32.105:43028 Call#694292838 Retry#0: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/ 724a04f162c946f2 _a85c6f0bcd88e99f/. 724a04f162c946f2 -a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/ 724a04f162c946f2 -a85c6f0bcd88e9a9_1910094467_data.3.parq (inode 249154603): File does not exist. Holder DFSClient_NONMAPREDUCE_-569971520_1 does not have any open files.   ... View more

Hi, Is there a way to configure impala to leave the hdfs directory permission/ownership unchanged after issuing an "insert overwrite" command?   insert overwrite change the directory owner to impala .   Thanks in advance    ... View more

Hi,   Every now and then, we will get a file not found exception when impala is trying to read a parquet file that was deleted hours ago.   The impala table utilizing these parquet files was refreshed several times in between the time the parquet files was removed and the time the query failed with the file not found exception.   Is this an impala issue or namenode ?   Stack trace below. Thanks in advance.   java.io.FileNotFoundException: File does not exist: /bigdata/parquet/cmdb/cmdb_node_stats_1458182957292.parquet.snappy at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1929) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:558) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)   at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1215) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1203) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1193) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265) at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:257) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1492) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /bigdata/parquet/cmdb/cmdb_node_stats_1458182957292.parquet.snappy at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1929) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:558) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)   at org.apache.hadoop.ipc.Client.call(Client.java:1468) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213) ... 10 more     ... View more

I think this would come as no surprise but I am a little intrigued by the difference.   When running the same queries on uncompacted paritions i.e. 6k parquet files vs compacted one , 4 files (but same amount of data in both cases) the query time is 6x slower on the uncompacted partitions.   Now, I woud have thought that the only real diference is in the hdfs scan part, where it should be a litte slower but dot 6x slower (in my opinion). The other interesting fact is that the query on the compacted partition has  PerHostPeakMemUsage  and  PeakMemoryUsage larger than the same query on many files...   Would anyone know what is going on and if there is a way to improve this? The usage scenario here is that we use impala to query in near real time but we compact partitions with a 2 hour lag. Consequently, all queries on the last 2 hours are slow while queries on older data over 3 hour old are much faster.   NOte: stats are not computed on the uncompacted  partitions... that could explain it perhaps?   Uncompacted query fragent: split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:32s299ms max:33s144ms mean: 32s800ms stddev:305.458736.458.736335ms execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 12 AsyncTotalTime: 0ns AverageThreadTokens: 1.00 InactiveTotalTime: 0ns PeakMemoryUsage: 10.7 MiB PerHostPeakMemUsage: 64.1 MiB PrepareTime: 548ms RowsProduced: 30 TotalCpuTime: 1.39s TotalNetworkReceiveTime: 31.96s TotalNetworkSendTime: 148.17us TotalStorageWaitTime: 0ns TotalTime: 32.51s   Compacted query fragment: split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:4s331ms max:5s211ms mean: 4s854ms stddev:327.747516.747.515651ms execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 12 AsyncTotalTime: 0ns AverageThreadTokens: 1.00 InactiveTotalTime: 0ns PeakMemoryUsage: 18.8 MiB PerHostPeakMemUsage: 84.6 MiB PrepareTime: 521ms RowsProduced: 30 TotalCpuTime: 1.05s TotalNetworkReceiveTime: 4.33s TotalNetworkSendTime: 154.35us TotalStorageWaitTime: 0ns TotalTime: 4.86s   ... View more

Hi,   We recently increased the number of datanodes while kept the number of impala daemons constant. We now have 2x more datanode than impala daemons.     We had a hard time with impala while hdfs was rebalancing.  My question is , what are the cons of NOT having impala running on ALL datanodes? ... View more

  Hi, WE have been getting these file does not exist exceptions as show in the stact trace below. These files did exist at some point but didn't when the exception occured. Is this impala or namenode problem and why would this occur?? Thanks in advance.   hdfsOpenFile(hdfs://nameservice1/bigdata/parquet/mysql_table_size/year=2016/month=2/day=16/1455644846091.parquet.snappy): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error: java.io.FileNotFoundException: File does not exist: /bigdata/parquet/mysql_table_size/year=2016/month=2/day=16/1455644846091.parquet.snappy at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1929) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:558) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1215) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1203) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1193) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265) at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:257) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1492) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) ... View more

HI, we started having an OOM with the catalog .   E1203 15:00:07.379981 24194 catalog-server.cc:279] OutOfMemoryError: Requested array size exceeds VM limit   We get this every 4 sec. WE restarted catalog and it works fine for a while and then the error start reoccurring.   Our biggest table has 5k partitions and less than 20k files. Catalogd is using between 15-18GB of memory and we are on impala 2.2 .   Have we hit a hard limit here?    From the name node: 1037386 files and directories, 752946 blocks = 1790332 total filesystem object(s).   We have plan to grow our data by many folds... hope we are not capped.... Any help would be appreciated.           ... View more

Hi, We have had several daemon restarts and the last log entry before the restart was:   I1125 02:07:05.337430 32232 status.cc:114] File hdfs://nameservice1/bigdata/parquet/cmdb/cmdb_node_stats_1448445780889.parquet.snappy is invalid. Missing metadata. @ 0x78b793 (unknown) @ 0xbede4b (unknown) @ 0xbee501 (unknown) @ 0xbbb5aa (unknown) @ 0xaa4faa (unknown) @ 0xaa7130 (unknown) @ 0xca79b3 (unknown) @ 0x7f5ef0d3e9d1 start_thread @ 0x7f5eefc9c9dd clone     Would this error be responsible for the restart? (Not too obvious looking at the code)   Would a refresh table cmd executed while the parquet table was being written and be incomplete cause this restart? All indications points to this but would like a second opinion 🙂 Thanks in advance! Stephane ... View more