Thank you for sharing your information. Took me a while to figure out why I'm getting the legacy behaviour even though I'm on Spark 1.6.0 which is supposed to have unified memory management (CDH 5.10.1). Now my question is, does setting spark.memory.useLegacyMode to false have any unintended consequences?! Or Is it safe to simply set it to false?! And when does Cloudera plan to make it default?!
... View more