Member since
01-04-2016
409
Posts
313
Kudos Received
35
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 5013 | 01-16-2018 07:00 AM | |
| 1546 | 09-13-2017 06:17 PM | |
| 3126 | 09-13-2017 05:58 AM | |
| 1976 | 08-28-2017 07:16 AM | |
| 3845 | 05-11-2017 11:30 AM |
12-27-2017
09:30 AM
I am using HDP 2.6 and spark 2.1. While running select * from table trough beeline. Connection sting !connect jdbc:hive2://xxx.xxxx.xxxx.xxx:10016 I am getting following error. Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (, executor 7): TaskResultLost (result lost from block manager)
Driver stacktrace: (state=,code=0)
Error: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 4 tasks (1230.8 MB) is bigger than spark.driver.maxResultSize (1024.0 MB) (state=,code=0)
Error: java.lang.OutOfMemoryError: GC overhead limit exceeded (state=,code=0)
WARN TransportChannelHandler: Exception in connection from ip-192-168-181-26.ca-central-1.compute.internal/192.168.181.26:42866
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:711)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:700)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:221)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:141)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:296)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:177)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:168)
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:129)
at io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
17/12/27 04:18:26 ERROR TransportResponseHandler: Still have 1 requests outstanding when connection from ip-192-168-181-26.ca-central-1.compute.internal/192.168.181.26:42866 is closed
17/12/27 04:18:26 ERROR RetryingBlockFetcher: Failed to fetch block taskresult_8, and will not retry (0 retries)
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:711)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:700)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:221)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:141)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:296)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:177)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:168)
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:129)
at io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
17/12/27 04:18:26 WARN BlockManager: Failed to fetch block after 1 fetch failures. Most recent failure cause:
org.apache.spark.SparkException: Exception thrown in awaitResult:
at org.apache.spark.util.ThreadUtils$.awaitResult(ThreadUtils.scala:194)
at org.apache.spark.network.BlockTransferService.fetchBlockSync(BlockTransferService.scala:104)
at org.apache.spark.storage.BlockManager.getRemoteBytes(BlockManager.scala:593)
at org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:82)
at org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$1.apply(TaskResultGetter.scala:63)
at org.apache.spark.scheduler.TaskResultGetter$$anon$3$$anonfun$run$1.apply(TaskResultGetter.scala:63)
at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1963)
at org.apache.spark.scheduler.TaskResultGetter$$anon$3.run(TaskResultGetter.scala:62)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException: Boxed Error
at scala.concurrent.impl.Promise$.resolver(Promise.scala:55)
at scala.concurrent.impl.Promise$.scala$concurrent$impl$Promise$$resolveTry(Promise.scala:47)
at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:244)
at scala.concurrent.Promise$class.complete(Promise.scala:55)
at scala.concurrent.impl.Promise$DefaultPromise.complete(Promise.scala:153)
at scala.concurrent.Promise$class.failure(Promise.scala:104)
at scala.concurrent.impl.Promise$DefaultPromise.failure(Promise.scala:153)
at org.apache.spark.network.BlockTransferService$$anon$1.onBlockFetchFailure(BlockTransferService.scala:95)
at org.apache.spark.network.shuffle.RetryingBlockFetcher$RetryingBlockFetchListener.onBlockFetchFailure(RetryingBlockFetcher.java:231)
at org.apache.spark.network.shuffle.OneForOneBlockFetcher.failRemainingBlocks(OneForOneBlockFetcher.java:123)
at org.apache.spark.network.shuffle.OneForOneBlockFetcher.access$300(OneForOneBlockFetcher.java:43)
at org.apache.spark.network.shuffle.OneForOneBlockFetcher$ChunkCallback.onFailure(OneForOneBlockFetcher.java:79)
at org.apache.spark.network.client.TransportResponseHandler.failOutstandingRequests(TransportResponseHandler.java:107)
at org.apache.spark.network.client.TransportResponseHandler.exceptionCaught(TransportResponseHandler.java:138)
at org.apache.spark.network.server.TransportChannelHandler.exceptionCaught(TransportChannelHandler.java:81)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:260)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:252)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:260)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:252)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:260)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:252)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)
at org.apache.spark.network.util.TransportFrameDecoder.exceptionCaught(TransportFrameDecoder.java:190)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:260)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:252)
at io.netty.channel.ChannelHandlerAdapter.exceptionCaught(ChannelHandlerAdapter.java:79)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:260)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:252)
at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1261)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:281)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:260)
at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:899)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:87)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:162)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:643)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:566)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:480)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:442)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
... 1 more
Caused by: java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:693)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:711)
at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:700)
at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:237)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:221)
at io.netty.buffer.PoolArena.allocate(PoolArena.java:141)
at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:296)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:177)
at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:168)
at io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:129)
at io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
Need help urgent. Thanks in Advance.
... View more
Labels:
- Labels:
-
Apache Spark
12-26-2017
11:09 AM
@srinivasa rao I have tried with following properties
set hive.compute.query.using.stats=true; set hive.stats.fetch.column.stats=true; set hive.stats.fetch.partition.stats=true; set hive.fetch.task.conversion=more; set hive.support.concurrency=false it is taking 3 min. for count.
... View more
12-26-2017
04:47 AM
@srinivasa rao Query is simple. select count(*) from rasdb.test; in this table 3800000 record.
... View more
12-21-2017
05:19 AM
@Sindhu I have tried with above values. Still taking time. There is aroung 38 lakhs records. Query ID = ec2-user_20171221001453_85ebceae-ab99-4a88-aadf-8f3eb6d05fdb
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1513772289178_0006, Tracking URL = http://ip-192-168-180-54.ca-central-1.compute.internal:8088/proxy/application_1513772289178_0006/
Kill Command = /usr/hdp/2.6.2.14-5/hadoop/bin/hadoop job -kill job_1513772289178_0006
Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1
2017-12-21 00:15:10,576 Stage-1 map = 0%, reduce = 0%
2017-12-21 00:16:11,542 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 192.03 sec
2017-12-21 00:16:34,173 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 260.89 sec
2017-12-21 00:16:38,289 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 270.02 sec
2017-12-21 00:16:45,514 Stage-1 map = 67%, reduce = 22%, Cumulative CPU 279.97 sec
2017-12-21 00:17:35,937 Stage-1 map = 100%, reduce = 22%, Cumulative CPU 330.59 sec
2017-12-21 00:17:36,967 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 332.77 sec
MapReduce Total cumulative CPU time: 5 minutes 32 seconds 770 msec
Ended Job = job_1513772289178_0006
MapReduce Jobs Launched:
Stage-Stage-1: Map: 3 Reduce: 1 Cumulative CPU: 332.77 sec HDFS Read: 322983 HDFS Write: 8 SUCCESS
Total MapReduce CPU Time Spent: 5 minutes 32 seconds 770 msec
OK
3778700
... View more
12-18-2017
11:40 AM
@Sandeep Nemuri Also getting error while connecting through with beeline. Coneecton getting established successfully , but when run the query getting following error. Error: java.lang.OutOfMemoryError: GC overhead limit exceeded (state=,code=0) Added following property spark.yarn.am.memory=1g spark.yarn.am.cores=1 Please help to resolve the issue.
... View more
12-18-2017
05:44 AM
@sindhu Can you please help on this. I tried to run analyze it takes around 217 second. There is aroung 38 lakh's records. analyze table schema.table compute statistics for columns; Query ID = ec2-user_20171218003632_b89c66b2-2484-41b3-8d11-d0559e2b3ff7 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=<number> In order to limit the maximum number of reducers: set hive.exec.reducers.max=<number> In order to set a constant number of reducers: set mapreduce.job.reduces=<number> Starting Job = job_1513235262783_0081, Tracking URL = http://ip-192-168-180-54.ca-central-1.compute.internal:8088/proxy/application_1513235262783_0081/ Kill Command = /usr/hdp/2.6.2.14-5/hadoop/bin/hadoop job -kill job_1513235262783_0081 Hadoop job information for Stage-0: number of mappers: 3; number of reducers: 1 2017-12-18 00:36:41,824 Stage-0 map = 0%, reduce = 0% 2017-12-18 00:37:42,572 Stage-0 map = 0%, reduce = 0%, Cumulative CPU 189.89 sec 2017-12-18 00:38:32,943 Stage-0 map = 33%, reduce = 0%, Cumulative CPU 352.13 sec 2017-12-18 00:38:37,056 Stage-0 map = 67%, reduce = 0%, Cumulative CPU 359.55 sec 2017-12-18 00:38:43,223 Stage-0 map = 67%, reduce = 22%, Cumulative CPU 366.29 sec 2017-12-18 00:39:43,898 Stage-0 map = 67%, reduce = 22%, Cumulative CPU 428.53 sec 2017-12-18 00:40:06,476 Stage-0 map = 100%, reduce = 22%, Cumulative CPU 454.93 sec 2017-12-18 00:40:07,503 Stage-0 map = 100%, reduce = 67%, Cumulative CPU 455.44 sec 2017-12-18 00:40:08,526 Stage-0 map = 100%, reduce = 100%, Cumulative CPU 457.46 sec MapReduce Total cumulative CPU time: 7 minutes 37 seconds 460 msec Ended Job = job_1513235262783_0081 MapReduce Jobs Launched: Stage-Stage-0: Map: 3 Reduce: 1 Cumulative CPU: 457.46 sec HDFS Read: 688303 HDFS Write: 2067 SUCCESS Total MapReduce CPU Time Spent: 7 minutes 37 seconds 460 msec OK Time taken: 216.908 seconds I tried to run count qury after analyze stil its take 5 min. Can you please help to tune hive so it work faster. We use map reduce engine. Query ID = ec2-user_20171218004512_6bef2ddb-d981-42f8-b2e5-c42a9ad80bfd
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Starting Job = job_1513235262783_0082, Tracking URL = http://ip-192-168-180-54.ca-central-1.compute.internal:8088/proxy/application_1513235262783_0082/
Kill Command = /usr/hdp/2.6.2.14-5/hadoop/bin/hadoop job -kill job_1513235262783_0082
Hadoop job information for Stage-1: number of mappers: 3; number of reducers: 1
2017-12-18 00:45:22,022 Stage-1 map = 0%, reduce = 0%
2017-12-18 00:46:22,804 Stage-1 map = 0%, reduce = 0%, Cumulative CPU 189.83 sec
2017-12-18 00:46:43,363 Stage-1 map = 33%, reduce = 0%, Cumulative CPU 251.69 sec
2017-12-18 00:46:46,436 Stage-1 map = 67%, reduce = 0%, Cumulative CPU 259.47 sec
2017-12-18 00:46:54,666 Stage-1 map = 67%, reduce = 22%, Cumulative CPU 269.17 sec
2017-12-18 00:47:49,101 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 328.79 sec
MapReduce Total cumulative CPU time: 5 minutes 28 seconds 790 msec
Ended Job = job_1513235262783_0082
MapReduce Jobs Launched:
Stage-Stage-1: Map: 3 Reduce: 1 Cumulative CPU: 328.79 sec HDFS Read: 330169 HDFS Write: 8 SUCCESS
Total MapReduce CPU Time Spent: 5 minutes 28 seconds 790 msec
OK
3778700
... View more