Hi @Sidhartha, The error means that the destination table base.customers already has the partition source_name=ORACLE, as the error indicated: Partition already exists [customers(source_name=ORACLE)] Can you run : SHOW PARTITIONS base.customers; to confirm? If yes, try to drop it and then run EXCHANGE PARTITION again. Cheers Eric ... View more

There is something very unusual happening here. Based on your outputs, values are not only ending up in the wrong columns, but you are even getting different values!   In the 'correct' record, you have 5686.76, and in the 'wrong' record you have -5686.76.   My first guess was that there is a mistake in how you send data to the appropriate columns, but I don't see how that can explain a minus sign changing position.   To troubleshoot something like this, it is really important to dig into the details. I would therefore recommend you to bring your question down to a 'Minimal reproducible example'. Eliminating any complexity that is not causing unexpected results.   For example: You show a load command to get data into spark, consider replacing it with an actual string (and make sure to check whether the string allows you to reproduce the problem).    You also show 2 writes, but if we have the exact input and code to reproduce the problem the correct answer is probably not relevant.   Also, you use some code to list columns, consider hardcoding it first.   As mentioned, really try to take out all complexity untill we land on a minimal amount that still reproduces the problem.   Hopefully you will already see the answer once you have eliminated all the distractions, and if not you will have a fully trimmed down version, which you can use to update your question here! ... View more

According to the error, it is looking for java 7 installed by cloudera. You should define JAVA_HOME={path_to_your_jdk8_installation} in bashrc. ... View more

The error complains about the value of "hadoop.security.authentication". You have set it to "Kerberos" while the accepted values are "simple" and "kerberos" (all letters in lowercase). ... View more

The mistake I made was Case class should be outside the main and inside the object In this line: val partfile = sparkSession.read.text("partfile").as[String] , I used  read.text("..") to get a file into Spark where we can use  read.textFile("...") ... View more