Reply
Expert Contributor
Posts: 132
Registered: ‎08-07-2017

memory leakage issue in spark

[ Edited ]

Hello All,

 

We are running spark application and memory usage is increasing continuously after every run. I think it is beacuse of memory leakage. Can you please provide pointers for troubleshooting?

 

Thanks,

Priya

Posts: 44
Topics: 0
Kudos: 6
Solutions: 4
Registered: ‎05-15-2018

Re: memory leakage issue in spark

Hello Priya,

 

Thanks for posting your query.

 

I see you have reported your spark application memory usage is getting increased on very run, and you suspect it is because of memory leakage

 

Could you please share where exactly (Driver,Executor) you are observing memory leakage

 

This will help us to capture certain diagnostic information 

Thanks,
Satz
Expert Contributor
Posts: 132
Registered: ‎08-07-2017

Re: memory leakage issue in spark

Hi satz,

Thanks for inputs.

Now we are getting below errors for spark thrift service.

ERROR curator.ConnectionState: Connection timed out for connection string (zookeeper servers:2181) and timeout (15000) / elapsed (654234)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:793)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:779)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:58)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:265)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


and

ERROR imps.CuratorFrameworkImpl: Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:695)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:496)
at org.apache.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:50)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

and
ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FAILED!

Please suggest.

Thanks,
Priya
Posts: 44
Topics: 0
Kudos: 6
Solutions: 4
Registered: ‎05-15-2018

Re: memory leakage issue in spark

Hello Priya,

 

I see from your initial thread that you are seeing Memory leaks on your application. Could you please help us with more details

 

Like, these messages which you are seeing as a side effect of your GC or memory leak ?

 

Basically, the error denotes that the zookeeper client connection is getting timed out.

 

Also, please look for any OutOfMemory or GC Overhead messages or please check from Zookeeper server side as well

Thanks,
Satz
Highlighted
Expert Contributor
Posts: 132
Registered: ‎08-07-2017

Re: memory leakage issue in spark

Hi Satz,

Thanks for inputs. Currently I don't see any OutOfMemory or GC Overhead messages.

I see messages like the one I mentioned above and in addition to that messages like below as well

ERROR server.TransportRequestHandler: Error sending result RpcResponse{requestId=6993342906751026461, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=81 cap=81]}} to datanodes; closing connection

Thanks,
Priya
Announcements