I am running insert into select .... query with dynamic partitions to insert the data into hive parquet table through spark-sql.
it takes huge time, when i see application master UI i see all the tasks of spark have been completed but dont see data in hive table , but after 30 mins i see data start appearing in hive table. strange ?
But if i run same query in IMPALA, it completes in no time.
Have seen there are many users who are facing same issue but have not got the solution. any solution please ?