Hey @bkosaraju , thanks for sharing your thoughts. I had this as one of the alternatives but looking is there any other way in Spark SQL like transaction level control (like commit or rollback) if a data frame is not created or for any exception. Thanks you. ... View more