Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is their any way of to use Spark memory instead of hard disk ?

Highlighted

Is their any way of to use Spark memory instead of hard disk ?

New Contributor

Hi,

I want to use spark memory instead of hard disk. So is their any way to run spark job on memory only.

I am getting space error in hard disk so want to run job on memory only not in hard disk.

2 REPLIES 2

Re: Is their any way of to use Spark memory instead of hard disk ?

1. User RDD.cache() and set the Policy to MEMORY_ONLY
2. Please ensure your executors have enough memory to hold your data completely in memory

Re: Is their any way of to use Spark memory instead of hard disk ?

New Contributor

Thanks @kgautam. I will try the same and let you know.