Reply
Highlighted
New Contributor
Posts: 1
Registered: ‎03-21-2018

Using --direct option in Sqoop import/export

[ Edited ]

Hi Cloudera team. 

I'm currently using Sqoop 1.4.6-cdh5.12.0. I've found strange behavior using sqoop import/export with --direct option. While trying to import table into hdfs using below sqoop import command, it works as expected:

sqoop import \
--connect jdbc:mysql://quickstart.cloudera/retail_db \
--username retail_dba \
--password cloudera \
--table categories \
--target-dir export_problem/categories \
--fields-terminated-by '\t' \
--direct

Result:

[cloudera@quickstart ~]$ hadoop fs -cat export_problem/categories/p* 
1	2	Football
2	2	Soccer
3	2	Baseball & Softball
4	2	Basketball
...

 

 

But when I try to export it back to mysql with the same --direct option it doesn't work as expected:

mysql> create table categories_export as select * from categories limit 0;
Query OK, 0 rows affected (0.02 sec)
Records: 0  Duplicates: 0  Warnings: 0
sqoop export \
--connect jdbc:mysql://quickstart.cloudera/retail_db \
--username retail_dba \
--password cloudera \
--export-dir export_problem/categories \
--table categories_export \
--input-fields-terminated-by "\t" \
--direct

Result:

 

mysql> select * from categories_export limit 5;
+-------------+------------------------+---------------+
| category_id | category_department_id | category_name |
+-------------+------------------------+---------------+
|           1 |                   NULL |               |
|           2 |                   NULL |               |
|           3 |                   NULL |               |
|           4 |                   NULL |               |
|           5 |                   NULL |               |
+-------------+------------------------+---------------+
5 rows in set (0.00 sec)

But without --direct option all works as expected:

sqoop export \
--connect jdbc:mysql://quickstart.cloudera/retail_db \
--username retail_dba \
--password cloudera \
--export-dir export_problem/categories \
--table categories_export \
--input-fields-terminated-by "\t" 

Result:

mysql> select * from categories_export limit 5;
+-------------+------------------------+---------------------+
| category_id | category_department_id | category_name       |
+-------------+------------------------+---------------------+
|           1 |                      2 | Football            |
|           2 |                      2 | Soccer              |
|           3 |                      2 | Baseball & Softball |
|           4 |                      2 | Basketball          |
|           5 |                      2 | Lacrosse            |
+-------------+------------------------+---------------------+
5 rows in set (0.00 sec)

Looks like in sqoop export --direct overwrites previous options but in sqoop import not.

Is it an expected behavior?

Cloudera Employee
Posts: 257
Registered: ‎03-23-2015

Re: Using --direct option in Sqoop import/export

Hi,

Thanks for reporting the issue, I will try to re-produce this in my lab and will let you know my findings.

Cheers
Announcements
New solutions