Welcome to the community @AtomJunkYarder . As this is an older post you would have a better chance of receiving a resolution by starting a new thread. This will also provide the opportunity to provide details specific to your environment that could aid others in providing a more accurate answer to your question.  ... View more

@Sidhartha Bobby To run exchange partition successfully, Your destination table base.customers cannot contain the partition that you are exchanging. But base.customers table is already have the partition(source_name=ORACLE) that you are exchanging. Resolution: Delete the existing partition in destination table and run the exchange partition command again. (or) Exchange the partition that is already not exists in destination table. If you want to just append the data to Destination table then run insert into base.customers table by selecting from stg.customers_testcontrol_staging table. From HiveDocs: Constraints for Hive Exchange partitions: The destination table cannot contain the partition to be exchanged. The operation fails in the presence of an index. Exchange partition is not allowed with transactional tables either as source or destination. Alternatively, use LOAD DATA or INSERT OVERWRITE commands to move partitions across transactional tables. This command requires both the source and destination table names to have the same table schema. If the schemas are different, the following exception is thrown: The tables have different schemas. Their partitions cannot be exchanged ... View more

With Hive 3 pushing hard with fully managed tables with native file formats as transactional tables, see https://docs.hortonworks.com/HDPDocuments/HDP3/HDP-3.1.0/managing-hive/content/hive_acid_operations.html for more info, this "direct from Spark to Hive" approach will get much harder do to the underlying "delta files" that get created when data is added/modified/removed from a Hive table. The Spark LLAP Connector will aid in this integration. That said, historically, the better answer is often to simple save your DF from Spark to HDFS, wrap it with an External Hive table and then do and INSERT INTO your existing Hive table with a SELECT * FROM your new external table. This lets Hive do all the heavy lifting and file conversions as needed and takes care of any partitioning and/or bucketing that you have in place. Good luck and happy Hadooping! ... View more