Member since
11-25-2015
16
Posts
1
Kudos Received
0
Solutions
10-17-2016
02:49 PM
we just hit the same issue... CDH 5.4 ... is there a know fix?
... View more
04-15-2016
11:33 AM
Thanks for the help. 4min for parquet encoding (800mb) seems high. Luckily its infrequent. We are on impala 2.2 . Is this something that got improved in more recent version?
... View more
04-14-2016
10:30 AM
Thanks for quick help. One last question . This snippet is frmo the same query as my initla question and I am wondering if this is the cause. EncodeTimer of 4min ... Does this rpresent the time taken to encode to parquet? And could it be where the slow down is? HdfsTableSink (4.4m) AsyncTotalTime: 0ns BytesWritten: 875.0 MiB CompressTimer: 9.18s EncodeTimer: 4.1m FilesCreated: 4 FinalizePartitionFileTimer: 4.02s HdfsWriteTimer: 3.93s InactiveTotalTime: 0ns PartitionsCreated: 1 PeakMemoryUsage: 319.9 MiB RowsInserted: 3,720,826 TmpFileCreateTimer: 55ms TotalTime: 4.4m
... View more
04-13-2016
11:23 AM
We are one specific query, "insert overwrite select * from a partitions" that usually take < 2min. Occasionaly , it takes over 5 min for unknown reason. Looking at the query profile, we see the following below. Throughput of < then 600kb sec is unexpected since we have 10GB network... So what could explain this? Is it releated to ThriftTransmitTime? Thanks DataStreamSender (dst_id=1) (5.0m) AsyncTotalTime: 0ns BytesSent: 169.9 MiB InactiveTotalTime: 0ns NetworkThroughput(*): 595.0 KiB/s OverallThroughput: 585.7 KiB/s PeakMemoryUsage: 72.0 KiB SerializeBatchTime: 2.77s ThriftTransmitTime(*): 4.9m TotalTime: 5.0m UncompressedRowBatchSize: 547.1 MiB
... View more
04-05-2016
09:55 AM
1 Kudo
The following query is getting hdfs error: insert overwrite bigdata.nodestats_swap partition (year,month,day,hour) select * from bigdata.nodestats where year=2016 and month=4 and day=5 and hour=11 It does not happen very often but would like to understand why\what happens. Below are the imala and namenode logs. Thanks in advance. Impala log: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /user/hive/warehouse/bigdata.db/ nodestats _swap/_impala_insert_staging/724a04f162c946f2_a85c6f0bcd88e99f/.724a04f162c946f2-a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/724a04f162c946f2-a85c6f0bcd88e9a9_1910094467_data.3.parq (inode 249154603): File does not exist. Holder DFSClient_NONMAPREDUCE_-569971520_1 does not have any open files. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3602) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:3399) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3255) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:667) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.addBlock(AuthorizationProviderProxyClientProtocol.java:212) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:483) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at org.apache.hadoop.ipc.Client.call(Client.java:1468) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy14.addBlock(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.addBlock(ClientNamenodeProtocolTranslatorPB.java:399) at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy15.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.locateFollowingBlock(DFSOutputStream.java:1544) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:600) I0405 06:24:37.250735 19067 runtime-state.cc:230] Error from query 724a04f162c946f2:a85c6f0bcd88e99f: Failed to close HDFS file: hdfs://nameservice1/user/hive/warehouse/bigdata.db/ nodestats _swap/_impala_insert_staging/724a04f162c946f2_a85c6f0bcd88e99f//.724a04f162c946f2-a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/724a04f162c946f2-a85c6f0bcd88e9a9_1910094467_data.3.parq Error(255): Unknown error 255 Relevant Namenode logs: 2016-04-05 06:23:39,449 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* allocateBlock: /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/ 724a04f162c946f2 _a85c6f0bcd88e99f/. 724a04f162c946f2 -a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/ 724a04f162c946f2 -a85c6f0bcd88e9a9_1910094467_data.2.parq. BP-1123871397-10.216.32.10-1433806145781 blk_1190477181_116736497{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-caeb90ae-cc3d-4142-9ac1-d83290ba9d4a:NORMAL:10.216.32.105:50010|RBW], ReplicaUnderConstruction[[DISK]DS-2a57f3a9-b316-481f-a3a8-d2430f07dba5:NORMAL:10.216.34.101:50010|RBW], ReplicaUnderConstruction[[DISK]DS-f90cdf95-8c43-4e81-ad37-d2c0dc384634:NORMAL:10.216.34.105:50010|RBW]]} 2016-04-05 06:23:40,770 INFO org.apache.hadoop.hdfs.StateChange: DIR* completeFile: /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/ 724a04f162c946f2 _a85c6f0bcd88e99f/. 724a04f162c946f2 -a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/ 724a04f162c946f2 -a85c6f0bcd88e9a9_1910094467_data.2.parq is closed by DFSClient_NONMAPREDUCE_-569971520_1 2016-04-05 06:24:37,163 WARN org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:impala (auth:SIMPLE) cause:org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/ 724a04f162c946f2 _a85c6f0bcd88e99f/. 724a04f162c946f2 -a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/ 724a04f162c946f2 -a85c6f0bcd88e9a9_1910094467_data.3.parq (inode 249154603): File does not exist. Holder DFSClient_NONMAPREDUCE_-569971520_1 does not have any open files. 2016-04-05 06:24:37,163 INFO org.apache.hadoop.ipc.Server: IPC Server handler 249 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 10.216.32.105:43028 Call#694292838 Retry#0: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/hive/warehouse/bigdata.db/nodestats_swap/_impala_insert_staging/ 724a04f162c946f2 _a85c6f0bcd88e99f/. 724a04f162c946f2 -a85c6f0bcd88e9a9_1469794349_dir/year=2016/month=4/day=5/hour=11/ 724a04f162c946f2 -a85c6f0bcd88e9a9_1910094467_data.3.parq (inode 249154603): File does not exist. Holder DFSClient_NONMAPREDUCE_-569971520_1 does not have any open files.
... View more
04-01-2016
09:53 AM
I see this was addressed as patch for IMPALA-827 but it only works for insert statements . The code is such that insert_inherit_permissions is not checked for insert OVERWRITE... is this by design or oversight?
... View more
03-30-2016
07:50 AM
We have been running for over 2 year with this set up , wihout invalidate. We simply refresh the table to pick up new parquet files and we have been fine all this time. Except for a handful of times (like less than 10 times in 2 years). I am thinking updates to metastore/catelog services either fail silently or are extremely slow at times.
... View more
03-25-2016
12:47 PM
Hi, Is there a way to configure impala to leave the hdfs directory permission/ownership unchanged after issuing an "insert overwrite" command? insert overwrite change the directory owner to impala . Thanks in advance
... View more
03-24-2016
10:18 AM
Hi, Every now and then, we will get a file not found exception when impala is trying to read a parquet file that was deleted hours ago. The impala table utilizing these parquet files was refreshed several times in between the time the parquet files was removed and the time the query failed with the file not found exception. Is this an impala issue or namenode ? Stack trace below. Thanks in advance. java.io.FileNotFoundException: File does not exist: /bigdata/parquet/cmdb/cmdb_node_stats_1458182957292.parquet.snappy at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1929) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:558) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1215) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1203) at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1193) at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299) at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265) at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:257) at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1492) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302) at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298) at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.FileNotFoundException): File does not exist: /bigdata/parquet/cmdb/cmdb_node_stats_1458182957292.parquet.snappy at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66) at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1929) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:558) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038) at org.apache.hadoop.ipc.Client.call(Client.java:1468) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) at com.sun.proxy.$Proxy14.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:254) at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at com.sun.proxy.$Proxy15.getBlockLocations(Unknown Source) at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1213) ... 10 more
... View more
03-23-2016
10:32 AM
I think this would come as no surprise but I am a little intrigued by the difference. When running the same queries on uncompacted paritions i.e. 6k parquet files vs compacted one , 4 files (but same amount of data in both cases) the query time is 6x slower on the uncompacted partitions. Now, I woud have thought that the only real diference is in the hdfs scan part, where it should be a litte slower but dot 6x slower (in my opinion). The other interesting fact is that the query on the compacted partition has PerHostPeakMemUsage and PeakMemoryUsage larger than the same query on many files... Would anyone know what is going on and if there is a way to improve this? The usage scenario here is that we use impala to query in near real time but we compact partitions with a 2 hour lag. Consequently, all queries on the last 2 hours are slow while queries on older data over 3 hour old are much faster. NOte: stats are not computed on the uncompacted partitions... that could explain it perhaps? Uncompacted query fragent: split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:32s299ms max:33s144ms mean: 32s800ms stddev:305.458736.458.736335ms execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 12 AsyncTotalTime: 0ns AverageThreadTokens: 1.00 InactiveTotalTime: 0ns PeakMemoryUsage: 10.7 MiB PerHostPeakMemUsage: 64.1 MiB PrepareTime: 548ms RowsProduced: 30 TotalCpuTime: 1.39s TotalNetworkReceiveTime: 31.96s TotalNetworkSendTime: 148.17us TotalStorageWaitTime: 0ns TotalTime: 32.51s Compacted query fragment: split sizes: min: 0, max: 0, avg: 0, stddev: 0 completion times: min:4s331ms max:5s211ms mean: 4s854ms stddev:327.747516.747.515651ms execution rates: min:0.00 /sec max:0.00 /sec mean:0.00 /sec stddev:0.00 /sec num instances: 12 AsyncTotalTime: 0ns AverageThreadTokens: 1.00 InactiveTotalTime: 0ns PeakMemoryUsage: 18.8 MiB PerHostPeakMemUsage: 84.6 MiB PrepareTime: 521ms RowsProduced: 30 TotalCpuTime: 1.05s TotalNetworkReceiveTime: 4.33s TotalNetworkSendTime: 154.35us TotalStorageWaitTime: 0ns TotalTime: 4.86s
... View more
03-23-2016
10:23 AM
The problem was caused by defective disks on the new servers. When daemon was using data on these defectve disks, the side effect was that queries were running extremey slow mainly due to the fact that these disks performance we 10x slower than they should have been. Once we took that server offline, things came back to normal.
... View more
03-03-2016
02:09 PM
thanks for the great explanation. We are trying to see if there is a link between the number of namenode without impala daemon and a very slow metastore update. AS shown below step #7 is worrysome. Could there be a link? Start execution: 35.57us (35.57us) Planning finished: 18ms (18ms) Ready to start remote fragments: 22ms (4ms) Remote fragments started: 46ms (23ms) DML data written: 5.08s (5.03s) Cancelled: 5.7m (5.6m) DML Metastore update finished: 16.9m (11.3m) Request finished: 16.9m (1ms) Unregister query: 16.9m (276ms)
... View more
03-03-2016
11:19 AM
Hi, We recently increased the number of datanodes while kept the number of impala daemons constant. We now have 2x more datanode than impala daemons. We had a hard time with impala while hdfs was rebalancing. My question is , what are the cons of NOT having impala running on ALL datanodes?
... View more
02-17-2016
03:31 PM
Hi, WE have been getting these file does not exist exceptions as show in the stact trace below. These files did exist at some point but didn't when the exception occured. Is this impala or namenode problem and why would this occur?? Thanks in advance. hdfsOpenFile(hdfs://nameservice1/bigdata/parquet/mysql_table_size/year=2016/month=2/day=16/1455644846091.parquet.snappy): FileSystem#open((Lorg/apache/hadoop/fs/Path;I)Lorg/apache/hadoop/fs/FSDataInputStream;) error:
java.io.FileNotFoundException: File does not exist: /bigdata/parquet/mysql_table_size/year=2016/month=2/day=16/1455644846091.parquet.snappy
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1929)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1870)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1850)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1822)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:558)
at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73)
at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1215)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1203)
at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1193)
at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:299)
at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:265)
at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:257)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1492)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:302)
at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:298)
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:298)
... View more
12-03-2015
05:12 PM
HI, we started having an OOM with the catalog . E1203 15:00:07.379981 24194 catalog-server.cc:279] OutOfMemoryError: Requested array size exceeds VM limit We get this every 4 sec. WE restarted catalog and it works fine for a while and then the error start reoccurring. Our biggest table has 5k partitions and less than 20k files. Catalogd is using between 15-18GB of memory and we are on impala 2.2 . Have we hit a hard limit here? From the name node: 1037386 files and directories, 752946 blocks = 1790332 total filesystem object(s). We have plan to grow our data by many folds... hope we are not capped.... Any help would be appreciated.
... View more
11-25-2015
09:23 AM
Hi, We have had several daemon restarts and the last log entry before the restart was: I1125 02:07:05.337430 32232 status.cc:114] File hdfs://nameservice1/bigdata/parquet/cmdb/cmdb_node_stats_1448445780889.parquet.snappy is invalid. Missing metadata. @ 0x78b793 (unknown) @ 0xbede4b (unknown) @ 0xbee501 (unknown) @ 0xbbb5aa (unknown) @ 0xaa4faa (unknown) @ 0xaa7130 (unknown) @ 0xca79b3 (unknown) @ 0x7f5ef0d3e9d1 start_thread @ 0x7f5eefc9c9dd clone Would this error be responsible for the restart? (Not too obvious looking at the code) Would a refresh table cmd executed while the parquet table was being written and be incomplete cause this restart? All indications points to this but would like a second opinion 🙂 Thanks in advance! Stephane
... View more
- Tags:
- restart