/usr/bin/hadoop: Argument list too long during hadoop fs -put

[#1]

I have around 1 million small files that i wanted to transfer from local to hdfs. I am getting the below error.

Command: hadoop fs -put path/* hdfspath/

/usr/bin/hadoop: Argument list too long

Options tried

1. Increased GCOverheadLimit

2. increased ulimit -s to 102 mb.

None of the options were successful. Can anyone please assist me?

[#2]

Can i get any help on this?

[#3]

Hi @Praneender Vuppala

I haven't seen that error before but I bet you can probably get around with it by using find & xargs.
For example can you give this a try?

find /your/dir -name '*.txt' -print0|xargs -0 -P 4 -I % hadoop fs -put % /your/hdfs/destination

Let me know if that helps. If it resolves your problem, please take a moment to log in and click accept answer 🙂

PS. You can tweak with the -P flag to increase/decrease performance using parallelism. More described here.