I want to use spark memory instead of hard disk. So is their any way to run spark job on memory only.
I am getting space error in hard disk so want to run job on memory only not in hard disk.
1. User RDD.cache() and set the Policy to MEMORY_ONLY2. Please ensure your executors have enough memory to hold your data completely in memory
Thanks @kgautam. I will try the same and let you know.