- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
how can import all tables from sql server database tables to hbase using single sqoop import command? is it possible?
- Labels:
-
Apache HBase
Created ‎05-23-2016 10:58 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Created ‎05-23-2016 10:16 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There is no way to tell Sqoop to import all tables into HBase, becuase you have to use "--hbase-table" which is incompatible with "--import-all-tables". Note that HBase is not a general purpose data-base/storage, it's used to store a relatively small number of tables and provide real-time access to them, so it doesn't make sense to import 100s of tables into HBase. For a reasonably small number of tables you can create a script:
for t in t1 t2 t3; do sqoop --connect jdc:mysql://... --table $t --hbase-table $t --hbase-create-table ... done
Note that it's a good idea to pre-create HBase tables, for example to set splitting and compression etc, because Sqoop will not do that. Another approach for your project can be to import all your tables into Hive, create a few Hive tables mapped onto HBase, and populate them using your Hive imported tables.
Created ‎05-23-2016 11:07 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
can you try with below command
sqoop import-all-tables --connect jdbc:mysql://mysql_server/mysql_table --table mysql_table_name --hbase-table hbase_table_name --column-family metadata --hbase-create-table --username root -P
Created ‎05-23-2016 08:58 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Sqoop doesn’t now permit you to import, all at once, a relational table directly into an HBase table having multiple column families. To work around this limitation, you create the HBase table first and then execute Sqoop import operations to finish the task.
Created ‎05-23-2016 10:16 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
There is no way to tell Sqoop to import all tables into HBase, becuase you have to use "--hbase-table" which is incompatible with "--import-all-tables". Note that HBase is not a general purpose data-base/storage, it's used to store a relatively small number of tables and provide real-time access to them, so it doesn't make sense to import 100s of tables into HBase. For a reasonably small number of tables you can create a script:
for t in t1 t2 t3; do sqoop --connect jdc:mysql://... --table $t --hbase-table $t --hbase-create-table ... done
Note that it's a good idea to pre-create HBase tables, for example to set splitting and compression etc, because Sqoop will not do that. Another approach for your project can be to import all your tables into Hive, create a few Hive tables mapped onto HBase, and populate them using your Hive imported tables.
