best option to run a query on hadoop cluster A and import that data in hadoop cluster B

[#1]

Hi,

We have 2 different hadoop clusters(cloudera & hdp) , i have to run an import query on Impala twce a day to import that data into our hadoop cluster. What is the best way to do this? I see sqoop only imports from relational db, may be distcp but what if i want to do a query with where conditions?

Is there any other way? if i decide to get all data from partition using distcp anyways, wghat ports should be open on both clusters?

Thanks in advance.

[#2]

@PJ

You can try this!

insert overwrite directory "hdfs://<hdpClusterNameNodeHost>:<NNPort>/<yourDirStructure>/" <your query here>

Let know if that works for you.

[#3]

@Rahul Soni

thanks for the reply but i get this error:

hive> insert overwrite directory "hdfs://nn:8020/test/" select * from table;

FAILED: SemanticException Error creating temporary folder on: hdfs://nn:8020/test

[#4]

Sweet! Probably the user doesn't have permission to write on that cluster than 🙂

Can you please confirm?

[#5]

@Rahul Soni

I am using same exact user for both clusters, this user has all permissions.