Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

memory leakage issue in spark

memory leakage issue in spark

Contributor

Hello All,

 

We are running spark application and memory usage is increasing continuously after every run. I think it is beacuse of memory leakage. Can you please provide pointers for troubleshooting?

 

Thanks,

Priya

4 REPLIES 4

Re: memory leakage issue in spark

Rising Star

Hello Priya,

 

Thanks for posting your query.

 

I see you have reported your spark application memory usage is getting increased on very run, and you suspect it is because of memory leakage

 

Could you please share where exactly (Driver,Executor) you are observing memory leakage

 

This will help us to capture certain diagnostic information 

Thanks,
Satz

Re: memory leakage issue in spark

Contributor
Hi satz,

Thanks for inputs.

Now we are getting below errors for spark thrift service.

ERROR curator.ConnectionState: Connection timed out for connection string (zookeeper servers:2181) and timeout (15000) / elapsed (654234)
org.apache.curator.CuratorConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.curator.ConnectionState.checkTimeouts(ConnectionState.java:198)
at org.apache.curator.ConnectionState.getZooKeeper(ConnectionState.java:88)
at org.apache.curator.CuratorZookeeperClient.getZooKeeper(CuratorZookeeperClient.java:115)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.performBackgroundOperation(CuratorFrameworkImpl.java:793)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.backgroundOperationsLoop(CuratorFrameworkImpl.java:779)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.access$400(CuratorFrameworkImpl.java:58)
at org.apache.curator.framework.imps.CuratorFrameworkImpl$4.call(CuratorFrameworkImpl.java:265)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)


and

ERROR imps.CuratorFrameworkImpl: Background operation retry gave up
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.checkBackgroundRetry(CuratorFrameworkImpl.java:695)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:496)
at org.apache.curator.framework.imps.BackgroundSyncImpl$1.processResult(BackgroundSyncImpl.java:50)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:609)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)

and
ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FAILED!

Please suggest.

Thanks,
Priya

Re: memory leakage issue in spark

Rising Star

Hello Priya,

 

I see from your initial thread that you are seeing Memory leaks on your application. Could you please help us with more details

 

Like, these messages which you are seeing as a side effect of your GC or memory leak ?

 

Basically, the error denotes that the zookeeper client connection is getting timed out.

 

Also, please look for any OutOfMemory or GC Overhead messages or please check from Zookeeper server side as well

Thanks,
Satz
Highlighted

Re: memory leakage issue in spark

Contributor
Hi Satz,

Thanks for inputs. Currently I don't see any OutOfMemory or GC Overhead messages.

I see messages like the one I mentioned above and in addition to that messages like below as well

ERROR server.TransportRequestHandler: Error sending result RpcResponse{requestId=6993342906751026461, body=NioManagedBuffer{buf=java.nio.HeapByteBuffer[pos=0 lim=81 cap=81]}} to datanodes; closing connection

Thanks,
Priya