Is there a maximum volume of data that can be imported using SQOOP?

[#1]

Hi,

I'm looking to migrate 15 terrabytes of data into Hadoop and considereing FTP or SQOOP. Can anyone advise on the maximum volumes that SQOOP can handle as I've been told that its not normally used above 10Gb.

Thanks

Leigh

[#2]

@Leigh Perkins

The main question is "What is the source of data?"

if it's RDBMS then sqoop and answer is Yes..You can leverage sqoop to load 15TB of data

If it's not RDBMS then you should look into NiFi or Flume or if you just want to load data into HDFS then webhdfs

[#3]

Thanks. The migration is from Oracle so sounds like SQOOP will work fine.

[#4]

@Leigh Perkins Yes ..Now, most critical component is "Let your DBA know about this"

You are gold with Sqoop and Oracle marriage in this use case :)

[#5]

@Leigh Perkins Also, make sure that you designed your Hadoop cluster accordingly...storage and memory

http://www.slideshare.net/alxslva/effective-sqoop-best-practices-pitfalls-and-lessons-40370936

[#6]

@Leigh Perkins

limitation is on the database side not on sqoop

[#7]

Perfect, thanks.

[#8]

@Leigh Perkins make sure to limit batches otherwise you will kill your DB