Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Cluster of clusters?

Cluster of clusters?

New Contributor

Suppose I have more than one Hadoop cluster. Is there any way to run a MapReduce job (or Hive query) across the multiple clusters?

 

I might have more than one HDFS cluster for various admin or data organization reasons, but want to run a job that scans all the data. Perhaps there would be a small Hadoop cluster that is a front-end to the other (larger) clusters.

 

Has anyone heard of this? Does it makes sense?

 

Thank you,

Chuck

 

1 REPLY 1

Re: Cluster of clusters?

Master Guru
There's no existing method in Apache Hadoop currently that can encompass multiple MR clusters for a single job. You can, however, use multiple HDFS cluster's input URIs inside a single job.
Don't have an account?
Coming from Hortonworks? Activate your account here