Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how can import all tables from sql server database tables to hbase using single sqoop import command? is it possible?

avatar
Rising Star
 
1 ACCEPTED SOLUTION

avatar
Master Guru

There is no way to tell Sqoop to import all tables into HBase, becuase you have to use "--hbase-table" which is incompatible with "--import-all-tables". Note that HBase is not a general purpose data-base/storage, it's used to store a relatively small number of tables and provide real-time access to them, so it doesn't make sense to import 100s of tables into HBase. For a reasonably small number of tables you can create a script:

for t in t1 t2 t3; do
sqoop --connect jdc:mysql://... --table $t --hbase-table $t --hbase-create-table ...
done

Note that it's a good idea to pre-create HBase tables, for example to set splitting and compression etc, because Sqoop will not do that. Another approach for your project can be to import all your tables into Hive, create a few Hive tables mapped onto HBase, and populate them using your Hive imported tables.

View solution in original post

3 REPLIES 3

avatar

can you try with below command

sqoop import-all-tables --connect jdbc:mysql://mysql_server/mysql_table --table mysql_table_name --hbase-table hbase_table_name --column-family metadata --hbase-create-table --username root -P 

avatar
Explorer

Sqoop doesn’t now permit you to import, all at once, a relational table directly into an HBase table having multiple column families. To work around this limitation, you create the HBase table first and then execute Sqoop import operations to finish the task.

avatar
Master Guru

There is no way to tell Sqoop to import all tables into HBase, becuase you have to use "--hbase-table" which is incompatible with "--import-all-tables". Note that HBase is not a general purpose data-base/storage, it's used to store a relatively small number of tables and provide real-time access to them, so it doesn't make sense to import 100s of tables into HBase. For a reasonably small number of tables you can create a script:

for t in t1 t2 t3; do
sqoop --connect jdc:mysql://... --table $t --hbase-table $t --hbase-create-table ...
done

Note that it's a good idea to pre-create HBase tables, for example to set splitting and compression etc, because Sqoop will not do that. Another approach for your project can be to import all your tables into Hive, create a few Hive tables mapped onto HBase, and populate them using your Hive imported tables.