We have 2 versions of Spark, Spark1.6 and Spark2.0. 

The jobs in test cluster seem to be working fine but the jobs in production has issues with jars. It is somehow not taking the Spark2 jars in class path.

I have checked all the configs which are exactly same in both test and prod.

The current issue I'm facing is 

java.io.InvalidClassException: org.apache.commons.lang3.time.FastDateFormat; local class incompatible: stream classdesc serialVersionUID = 1, local class serialVersionUID = 2
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:616)

However I tried to overwrite the specific jar and I face another issue which also seems to be regarding the jar mismatch.

Please let me know how this can be resolved.