Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Is it possible to select a specific worker for a specific task in spark?

Is it possible to select a specific worker for a specific task in spark?

New Contributor

Actual requirement is like i have devices which is accessible through specific machines only, so is it possible that to assign a task which involved communication to a specific device to a worker from which that device is accessible ?

 

Assume that there is spark cluster with master and workers. Workers are machines which has access to specific devices, master has no access to those devices.

1 REPLY 1

Re: Is it possible to select a specific worker for a specific task in spark?

Expert Contributor

The closest you could get is using Node Labels (when it gets implemented in CDH). This feature in YARN lets you specify the worker nodes (NodeManagers) that you'd want the application containers to run on. However, it's not out yet and work is still going on to incorporate this.