Reply
Highlighted
Explorer
Posts: 58
Registered: ‎09-29-2016

insert into select statement with dynamic partitions takes huge time in sparl-sql

[ Edited ]

Hello Experts,

 

I am running insert into select .... query with dynamic partitions to insert the data into hive parquet table through spark-sql.

it takes huge time, when i see application master UI i see all the tasks of spark have been completed but dont see data in hive table , but after 30 mins i see data start appearing in hive table. strange ? 

 

But if i run same query in  IMPALA, it completes in no time.

 

Have seen there are many users who are facing same issue but have not got the solution. any solution please ?