@Shu, Thanks for responding. I tried to update ticket earlier but, couldn't happen. I am not sure if some configs which I had set were causing this issue but, after commenting those out it started working fine. Below are the configs:
.config("spark.memory.fraction", "0.3")
.config("spark.storage.memoryFraction", "0.8")
.config("spark.executor.heartbeatInterval", "300")
.config("spark.network.timeout", "900")
.config("spark.sql.autoBroadcastJoinThreshold", "-1")