08-07-2015 08:16 PM
Are there any recommendations to speed up deployment of app binaries to YARN?
I've been using RM REST APIs to submit apps to it with binaries located on HDFS. This tends to take a lot of time when the size of binaries to be deployed as YARN app are big in size (say, >500MB or more), and also when number of containers that I need are high.
I could probably speed this up by :
1. Turning off default 3 copies needed on HDFS
2. Using HDFS cluster-wide cache which can help avoid block reads
3. Using YARN resource localization
Do you have any recommendations which are definitely known to speed this up?
08-10-2015 07:15 PM