Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

calling Hadoop commands using python causes arguments list too long error

calling Hadoop commands using python causes arguments list too long error

New Contributor

Hi,

 

I have been following this thread to interact with my remote Hadoop cluster, I have created a util class that is called my multiple modules to execute hadoop commands using subprocess. The problem i am facing currently is that, it starts throwing the following exception after sometime.

 

 

 

OSError: [Errno 7] Argument list too long

 

 

 

 

here is how my subprocess call looks like:

 

 

 

args_list = ['/usr/bin/hdfs', 'dfs', '-copyToLocal', 'hdfs://namenode:port/path/to/file', 'localPath']
subprocess.Popen(args_list, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

 

 

 

 

Can someone please suggest how can i avoid this error?

 

Thanks for any help in advance.

 

Regards,

Wasif

 

Don't have an account?
Coming from Hortonworks? Activate your account here