question Re: Error loading a file into HDFS in Archives of Support Questions (Read Only)

Error loading a file into HDFS

jeremy_salazar — Fri, 16 Sep 2022 10:06:54 GMT

We are trying to uplaod a file to HDFS that is 80GB. We are writing the file to /opt/Hadoop/tmp/.hdfs-nfs. It works fine with small files, but not with larger ones.

Does anyone have an answer as to where the file should write in temporarly prior to it moving into HDFS?

Is there some sort of other setting we need to consider?

Re: Error loading a file into HDFS

aervits — Thu, 03 Mar 2016 01:00:39 GMT

@Jeremy Salazar you can upload to HDFS directly, if you're using NFS, it is not designed for larger files of your size, that's on the roadmap for NFSv4. I recommend to zip the file before you upload using -put command.

hdfs dfs -put file /user/username

Re: Error loading a file into HDFS

jeremy_salazar — Thu, 03 Mar 2016 03:31:42 GMT

Sorry I should have specified that we are using Ambari to upload into HDFS.

Re: Error loading a file into HDFS

aervits — Thu, 03 Mar 2016 03:34:49 GMT

@Jeremy Salazar you mean HDFS Ambari view? That won't work, it's too big of a file to upload via Ambari view. Consider using CLI.