I'm trying to implement the distributed workers scenario as described in the Cloudera Documentation (specifically "Using Workers for Machine Learning"), in Python. The idea is to define a queue of parameters in the main interactive session, and then launch distributed workers that read, pull values from the queue and consume them.
I tried implementing the scenario as:
import cdsw
global indexes
indexes = range(20)
code = "if len(indexes) > 0: element = indexes.pop(0) ; print(f'I am working on {element}!')"
workers = cdsw.launch_workers(n=2, cpu=0.2, memory=0.5, code=code)
However, this throws:
NameError: name 'indexes' is not defined
Indicating that the queue (indexes) is not visible by the workers.
Is there a solution/workaround for this issue?
... View more
