I have been following this thread to interact with my remote Hadoop cluster, I have created a util class that is called my multiple modules to execute hadoop commands using subprocess. The problem i am facing currently is that, it starts throwing the following exception after sometime.
OSError: [Errno 7] Argument list too long
here is how my subprocess call looks like:
args_list = ['/usr/bin/hdfs', 'dfs', '-copyToLocal', 'hdfs://namenode:port/path/to/file', 'localPath']
subprocess.Popen(args_list, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
Can someone please suggest how can i avoid this error?
Thanks for any help in advance.
... View more