Support Questions

Find answers, ask questions, and share your expertise

Is it possible to select a specific worker for a specific task in spark?

New Contributor

Actual requirement is like i have devices which is accessible through specific machines only, so is it possible that to assign a task which involved communication to a specific device to a worker from which that device is accessible ?


Assume that there is spark cluster with master and workers. Workers are machines which has access to specific devices, master has no access to those devices.


Expert Contributor

The closest you could get is using Node Labels (when it gets implemented in CDH). This feature in YARN lets you specify the worker nodes (NodeManagers) that you'd want the application containers to run on. However, it's not out yet and work is still going on to incorporate this.