Dataframe could not process same data like rdd. We tried lots of configuration about executor memory, driver memory , executormemoryoverhead and executor cores munbers . But We could not solve this memory problem.
According to spark and cloudera web page Dataframe better than RDD for memory and execution time. Also for memory error there are lots of answer in web ,but all answer for spark 2.x .
So we think that our spark version very old and we want to upgrade version 2.x.