Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Memory for RDD

Solved Go to solution
Highlighted

Memory for RDD

Hi All,

Couple of question , both are related to memory issue but in different situation

1. sc.textFile(file)

size of the file is 60 GB , whereas I dont have executor memory to load the file. How spark will behave now

2. val inputRDD = sc.textFile(file)

val mapRDD = inputRDD.map(_.split(","));

val DFRDD = mapRDD.map(p => Customer(p(0).trim.toInt, p(1), p(2), p(3), p(4))).toDF

at this line , cluster dont have memory , will it throw Out of Memory error ? or purge previous RDD to move further ?

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Memory for RDD

@Gobi Subramani spark keeps lineage for all the previous RDD in order to calculate the next RDD, not sure what the purge mean here but if there is no memory available then spark with throw oom and fail the job. as OOM is error so previous RDD is not recoverable in this case.

View solution in original post

3 REPLIES 3
Highlighted

Re: Memory for RDD

@Gobi Subramani spark keeps lineage for all the previous RDD in order to calculate the next RDD, not sure what the purge mean here but if there is no memory available then spark with throw oom and fail the job. as OOM is error so previous RDD is not recoverable in this case.

View solution in original post

Re: Memory for RDD

@Rajkumar Singh ,

when any action performed , inputRDD , mapRDD are evaluated and keep it in memory. when evaluating DFRDD , system knows that , there is no memory to perform this transformation , at this point, will spark remove inputRDD from memory to make a room for DFRDD ?

Highlighted

Re: Memory for RDD

@Gobi Subramani

in case of memory pressure Spark will automatically evict RDD partitions from Workers in an LRU manner if no caching or persistence applied. depending on worker memory available LRU eviction happens independently on each Worker node.

Don't have an account?
Coming from Hortonworks? Activate your account here