Hi, Can any one please clarify my understanding on the use case difference between 'SparkStandalone' and 'Spark on YARN' cluster. Spark Standalone Cluster: If we do not have huge volume of data to process. If number of nodes required to process data are something less than 10 nodes. Then good to go with Standalone cluster. Spark on YARN Cluster: If you have huge volume of data to process and had to use more number of nodes and hence you need a better cluster manager to manage these nodes. Then good to go with Spark on YARN cluster. Also can anyone please let me know the infrastructure specifications required for the 'Spark Standalone' cluster. For example in the case of 'Spark Standalone' if its having 10 Spark node cluster. Can we just have 1 reliable hardware/machine for cluster manager as a master node and rest of 9 machines as worker nodes as slave nodes? ... View more

Hi, Does Apache 'Spark Standalone' need HDFS? If it's required how Spark uses the HDFS block size during the Spark application execution. I mean am trying to understand what will be the HDFS role during Spark application execution. Spark documentation says that the processing parallelism is controlled through RDD partitions and the executors/cores. Can anyone please help me to understand. ... View more