Support Questions
Find answers, ask questions, and share your expertise
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Shared variable between distributed workers

Shared variable between distributed workers

New Contributor

I'm trying to implement the distributed workers scenario as described in the Cloudera Documentation (specifically "Using Workers for Machine Learning"), in Python. The idea is to define a queue of parameters in the main interactive session, and then launch distributed workers that read, pull values from the queue and consume them.

I tried implementing the scenario as:


import cdsw
global indexes
indexes = range(20)

code = "if len(indexes) > 0: element = indexes.pop(0) ; print(f'I am working on {element}!')"

workers = cdsw.launch_workers(n=2, cpu=0.2, memory=0.5, code=code)


 However, this throws:


NameError: name 'indexes' is not defined


Indicating that the queue (indexes) is not visible by the workers.

Is there a solution/workaround for this issue?

Don't have an account?
Coming from Hortonworks? Activate your account here