Thanks
@Mukesh Kumar for adding your answer.
However, I would like to add a note on the general aspects that can be looked up on for improving the performance.
As you rightly said, Parallelism is one important aspect.
We can adjust parallelism in code by increasing or decreasing the partitions -
- case where you have lot free cores to run your application and the number of partitions are comparatively less, you could increase the number of partitions and
- case where you derive an RDD from an existing RDD and you get a very small fraction of the parent RDD, you may not get any benefit in the child rdd having the same number of partitions as its parent, you can try reducing the number of partitions
Other aspects that you can consider are -
Data Locality
This refer to how near the data and code are.
There could be situations where there are no CPU cycles to start a task on local – spark can decide to
- WAIT - data movement not required
- Move over to a free CPU and start the task there – Data need to be moved
The wait time for CPU can be configured setting spark.locality.wait* properties. Based on the application, we can decide if waiting for more time saves us time compared to the data being shuffled across over the network.
Data Serialization
There are situations where framework may require to ship the data over the network or persist it.
In such scenarios, the objects are serialized. Java Serialization is used by default. However, serialization frameworks
like kryo has shown better results than the default serialization. The serialization can be set using spark.serializer
Memory Management
Another important aspect that you already stepped up on is Memory Management.
If the application
would require not to Persist/Cache the data, you could try reducing the Storage Memory fraction and increasing the
execution memory.
This is with the fact that the Execution Memory can evict the storage memory up to the configured threshold for Storage and the
reverse is not true.
We can adjust these memory value changing spark.memory.fraction and spark.memory.storageFraction
Also on a side note, all Java GC tuning methods could be applied to Spark Applications as well.
We can collect GC statistics using java options -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps
Also note that serializing helps reduce the GC overhead of fighting large number of smaller objects.
Finally, we could most importantly look at our code, some points to consider could be -
- Carry forward only data that is worth
ie. we could consider filtering, schema validation on structured datasets etc upfront before propogating them to the downstream logics/aggregations
- Know the data size - think before calling collect, use sample/subset for debug/testing
- Target the low hanging fruits - consider reduceByKey over groupByKey, this helps avoid lot data shuffle over the network
- Consider using broadcast variables for caching large read only variables
