Reply
New Contributor
Posts: 1
Registered: ‎04-05-2018

Is it possible to select a specific worker for a specific task in spark?

Actual requirement is like i have devices which is accessible through specific machines only, so is it possible that to assign a task which involved communication to a specific device to a worker from which that device is accessible ?

 

Assume that there is spark cluster with master and workers. Workers are machines which has access to specific devices, master has no access to those devices.

Cloudera Employee
Posts: 58
Registered: ‎11-16-2015

Re: Is it possible to select a specific worker for a specific task in spark?

The closest you could get is using Node Labels (when it gets implemented in CDH). This feature in YARN lets you specify the worker nodes (NodeManagers) that you'd want the application containers to run on. However, it's not out yet and work is still going on to incorporate this.

Announcements