Is there any specific method to know how much amount of memory we need to allocate for spark configuration like yan-utilities.py? Please suggest how can we tune spark?

[#1 · Accepted solution]

@Ram D

Nothing automated, however, you can configure Dynamic Resource Allocation manually, as one time activity:

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/config-dra-manual.htm...

Some more here:

https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_spark-guide/content/ch_tuning-spark.html