Support Questions
Find answers, ask questions, and share your expertise

Shared variable between distributed workers

New Contributor

I'm trying to implement the distributed workers scenario as described in the Cloudera Documentation (specifically "Using Workers for Machine Learning"), in Python. The idea is to define a queue of parameters in the main interactive session, and then launch distributed workers that read, pull values from the queue and consume them.

I tried implementing the scenario as:


import cdsw
global indexes
indexes = range(20)

code = "if len(indexes) > 0: element = indexes.pop(0) ; print(f'I am working on {element}!')"

workers = cdsw.launch_workers(n=2, cpu=0.2, memory=0.5, code=code)


 However, this throws:


NameError: name 'indexes' is not defined


Indicating that the queue (indexes) is not visible by the workers.

Is there a solution/workaround for this issue?