Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

insert into select statement with dynamic partitions takes huge time in sparl-sql

insert into select statement with dynamic partitions takes huge time in sparl-sql

Hello Experts,

 

I am running insert into select .... query with dynamic partitions to insert the data into hive parquet table through spark-sql.

it takes huge time, when i see application master UI i see all the tasks of spark have been completed but dont see data in hive table , but after 30 mins i see data start appearing in hive table. strange ? 

 

But if i run same query in  IMPALA, it completes in no time.

 

Have seen there are many users who are facing same issue but have not got the solution. any solution please ?

 

Don't have an account?
Coming from Hortonworks? Activate your account here