Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

/usr/bin/hadoop: Argument list too long during hadoop fs -put

Highlighted

/usr/bin/hadoop: Argument list too long during hadoop fs -put

Contributor

I have around 1 million small files that i wanted to transfer from local to hdfs. I am getting the below error.

Command: hadoop fs -put path/* hdfspath/
/usr/bin/hadoop: Argument list too long

Options tried

1. Increased GCOverheadLimit

2. increased ulimit -s to 102 mb.

None of the options were successful. Can anyone please assist me?

2 REPLIES 2

Re: /usr/bin/hadoop: Argument list too long during hadoop fs -put

Contributor

Can i get any help on this?

Re: /usr/bin/hadoop: Argument list too long during hadoop fs -put

Hi @Praneender Vuppala

I haven't seen that error before but I bet you can probably get around with it by using find & xargs.
For example can you give this a try?

find /your/dir -name '*.txt' -print0|xargs -0 -P 4 -I % hadoop fs -put % /your/hdfs/destination

Let me know if that helps. If it resolves your problem, please take a moment to log in and click accept answer :)

PS. You can tweak with the -P flag to increase/decrease performance using parallelism. More described here.