04-05-2018 06:00 AM
Actual requirement is like i have devices which is accessible through specific machines only, so is it possible that to assign a task which involved communication to a specific device to a worker from which that device is accessible ?
Assume that there is spark cluster with master and workers. Workers are machines which has access to specific devices, master has no access to those devices.
04-11-2018 03:33 AM
The closest you could get is using Node Labels (when it gets implemented in CDH). This feature in YARN lets you specify the worker nodes (NodeManagers) that you'd want the application containers to run on. However, it's not out yet and work is still going on to incorporate this.