memory leakage issue in spark

[#1]

Hello All,

We are running spark application and memory usage is increasing continuously after every run. I think it is beacuse of memory leakage. Can you please provide pointers for troubleshooting?

Thanks,

Priya

[#2]

Hello Priya,

Thanks for posting your query.

I see you have reported your spark application memory usage is getting increased on very run, and you suspect it is because of memory leakage

Could you please share where exactly (Driver,Executor) you are observing memory leakage

This will help us to capture certain diagnostic information 

[#3]

Hi satz,

Thanks for inputs.

Now we are getting below errors for spark thrift service.

ERROR curator.ConnectionState: Connection timed out for connection string (zookeeper servers:2181) and timeout (15000) / elapsed (654234)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:793)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:779)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:58)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:265)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


and

ERROR imps.CuratorFrameworkImpl: Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:695)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:496)
at org.apache.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:50)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

and
ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FAILED!

Please suggest.

Thanks,
Priya

[#4]

Hello Priya,

I see from your initial thread that you are seeing Memory leaks on your application. Could you please help us with more details

Like, these messages which you are seeing as a side effect of your GC or memory leak ?

Basically, the error denotes that the zookeeper client connection is getting timed out.

Also, please look for any OutOfMemory or GC Overhead messages or please check from Zookeeper server side as well

[#5]

Hi Satz,

Thanks for inputs. Currently I don't see any OutOfMemory or GC Overhead messages.

I see messages like the one I mentioned above and in addition to that messages like below as well

ERROR server.TransportRequestHandler: Error sending result RpcResponse{requestId=6993342906751026461, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=81 cap=81]}} to datanodes; closing connection

Thanks,
Priya