@harsh8 I think the answer is yes! Below I will try to demonstrate by creating a table from an existing dataset copied to HDFS [hdfs@bern sqoop]$ hdfs dfs -ls /tmp/sqoop
Found 1 items
-rw-r--r-- 3 hdfs hdfs 400 2021-06-29 10:14 /tmp/sqoop/hr.txt Contents of the file hr.txt [hdfs@bern sqoop]$ hdfs dfs -cat /tmp/sqoop/hr.txt
100,Geoffrey,manager,50000,Admin
101,Thomas,Oracle Consultant,15000,IT
102,Biden,Project Manager,28000,PM
103,Carmicheal,Bigdata developer,30000,BDS
104,Johnson,Treasurer,21000,Accounts
105,Gerald,Director,30000,Management
106,Paul,Director,30000,Management
105,Mark,CEO,90000,Management
105,Edward,Janitor,30000,Housing
105,Richard,Farm Manager,31000,Agriculture
105,Albert,System Engineer,21000,IT You MUST pre-create the table and database [root@bern sqoop]# mysql -uroot -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 123
Server version: 5.5.65-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> create database harsh8;
Query OK, 1 row affected (0.05 sec)
MariaDB [(none)]> use harsh8;
Database changed Pre-create the table to receive the datasets MariaDB [harsh8]> CREATE TABLE staff ( id INT NOT NULL PRIMARY KEY, Name VARCHAR(20), Position VARCHAR(20),Salary INT,Department VARCHAR(10));
Query OK, 0 rows affected (0.26 sec)
MariaDB [harsh8]> show tables;
+------------------+
| Tables_in_harsh8 |
+------------------+
| staff |
+------------------+
1 row in set (0.00 sec) Check the empty staff table structure MariaDB [harsh8]> describe staff;
+------------+-------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------+-------------+------+-----+---------+-------+
| id | int(11) | NO | PRI | NULL | |
| Name | varchar(20) | YES | | NULL | |
| Position | varchar(20) | YES | | NULL | |
| Salary | int(11) | YES | | NULL | |
| Department | varchar(10) | YES | | NULL | |
+------------+-------------+------+-----+---------+-------+
5 rows in set (0.14 sec) The empty table before the export MariaDB [harsh8]> select * from staff;
Empty set (0.00 sec) Run the export to import the HDFS data into the hasrh8.staff table [hdfs@bern sqoop]$ sqoop export \
--connect jdbc:mysql://localhost/harsh8 \
--username root \
--password 'w3lc0m31' \
--table staff \
--export-dir /tmp/sqoop/hr.txt Running sqoop job see the command snippet 21/06/29 10:23:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7.3.1.4.0-315
21/06/29 10:23:06 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/06/29 10:23:06 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
21/06/29 10:23:06 INFO tool.CodeGenTool: Beginning code generation
21/06/29 10:23:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `staff` AS t LIMIT 1
21/06/29 10:23:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `staff` AS t LIMIT 1
21/06/29 10:23:09 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/3.1.4.0-315/hadoop-mapreduce
21/06/29 10:23:22 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hdfs/compile/5097d7a0a163272ca680207ac06da7d5/staff.jar
21/06/29 10:23:22 INFO mapreduce.ExportJobBase: Beginning export of staff
21/06/29 10:25:10 INFO client.RMProxy: Connecting to ResourceManager at bern.swiss.ch/192.168.0.139:8050
21/06/29 10:25:13 INFO client.AHSProxy: Connecting to Application History server at bern.swiss.ch/192.168.0.139:10200
21/06/29 10:25:17 INFO mapreduce.JobResourceUploader: Disabling Erasure Coding for path: /user/hdfs/.staging/job_1624952474858_0001
21/06/29 10:25:50 INFO input.FileInputFormat: Total input files to process : 1
21/06/29 10:25:50 INFO input.FileInputFormat: Total input files to process : 1
21/06/29 10:25:52 INFO mapreduce.JobSubmitter: number of splits:4
21/06/29 10:25:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1624952474858_0001
21/06/29 10:25:57 INFO mapreduce.JobSubmitter: Executing with tokens: []
21/06/29 10:25:59 INFO conf.Configuration: found resource resource-types.xml at file:/etc/hadoop/3.1.4.0-315/0/resource-types.xml
21/06/29 10:26:01 INFO impl.YarnClientImpl: Submitted application application_1624952474858_0001
21/06/29 10:26:01 INFO mapreduce.Job: The url to track the job: http://bern.swiss.ch:8088/proxy/application_1624952474858_0001/
21/06/29 10:26:01 INFO mapreduce.Job: Running job: job_1624952474858_0001
21/06/29 10:29:36 INFO mapreduce.Job: Job job_1624952474858_0001 running in uber mode : false
21/06/29 10:29:36 INFO mapreduce.Job: map 0% reduce 0%
21/06/29 10:33:15 INFO mapreduce.Job: map 75% reduce 0%
21/06/29 10:33:16 INFO mapreduce.Job: map 100% reduce 0% YARN UI showing the job running and completing Completed successfully Data now uploaded in the destination table MariaDB [harsh8]> select * from staff;
+-----+------------+-------------------+--------+------------+
| id | Name | Position | Salary | Department |
+-----+------------+-------------------+--------+------------+
| 100 | Geoffrey | manager | 50000 | Admin |
| 101 | Thomas | Oracle Consultant | 15000 | IT |
| 102 | Biden | Project Manager | 28000 | PM |
| 103 | Carmicheal | Bigdata developer | 30000 | BDS |
| 104 | Johnson | Treasurer | 21000 | Accounts |
| 105 | Gerald | Director | 30000 | Management |
+-----+------------+-------------------+--------+------------+
6 rows in set (0.28 sec) I hope this answers your question
... View more
