Support Questions
Find answers, ask questions, and share your expertise

Is their any way of to use Spark memory instead of hard disk ?

Is their any way of to use Spark memory instead of hard disk ?

Explorer

Hi,

I want to use spark memory instead of hard disk. So is their any way to run spark job on memory only.

I am getting space error in hard disk so want to run job on memory only not in hard disk.

2 REPLIES 2
Highlighted

Re: Is their any way of to use Spark memory instead of hard disk ?

1. User RDD.cache() and set the Policy to MEMORY_ONLY
2. Please ensure your executors have enough memory to hold your data completely in memory

Highlighted

Re: Is their any way of to use Spark memory instead of hard disk ?

Explorer

Thanks @kgautam. I will try the same and let you know.