Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Moving .dat file into HDFS

Highlighted

Moving .dat file into HDFS

New Contributor

Hello Team,

I have .dat files ranging from 2GB - 500GB that are present in SAS server. I need to get these files and load it into Hive table. How can this be automated? I considered few options but they do not transfer the files fast.

Flume and Nifi: These are meant for realtime data ingestion. Though i can use them, but it will be an overkill. They are simply not meant for file transfer.

distcp: It is between HDFS cluster.On one server, i do not have HDFS

scp/winscp: These would be very slow over the network and for files >500G, I dont think anyone would recommend this approach.

Sqoop: I will not have access to tables, If i manage to get aceess, will i be able to transfer files by increasing number of mappers?

Any thoughts/suggestions are welcome and appreciated.

1 REPLY 1

Re: Moving .dat file into HDFS

New Contributor

Hello Experts,

can someone please help me?

Don't have an account?
Coming from Hortonworks? Activate your account here