Support Questions
Find answers, ask questions, and share your expertise

I want to import certain tables from multiple SQL sever databases using sqoop to HDFS. Can someone guide me how to do it? an automated script would do well.

Contributor
 
1 ACCEPTED SOLUTION

Hi @omkar pathallapalli, a single Sqoop command can import only a single table from a given DB server. So, to import multiple tables from multiple servers you need a command, for example a Bash script like this

for tbl in $(cat $4); do
sqoop import –-connect "jdbc:sqlserver://${1}:3464;databaseName=${2}" --username ${3} -P --table ${tbl} --target-dir sqimport
done

And call it once per DB server providing each DB server FQDN, database name, user-name and a file listing the tables you want to import (one table per line). The script will prompt you for the password. You can modify target-dir and/or add more Sqoop properties including the number of mappers used for import, by default 4.

View solution in original post

3 REPLIES 3

Super Guru

Hi @omkar pathallapalli, a single Sqoop command can import only a single table from a given DB server. So, to import multiple tables from multiple servers you need a command, for example a Bash script like this

for tbl in $(cat $4); do
sqoop import –-connect "jdbc:sqlserver://${1}:3464;databaseName=${2}" --username ${3} -P --table ${tbl} --target-dir sqimport
done

And call it once per DB server providing each DB server FQDN, database name, user-name and a file listing the tables you want to import (one table per line). The script will prompt you for the password. You can modify target-dir and/or add more Sqoop properties including the number of mappers used for import, by default 4.

Contributor

Nice solution @Predrag Minovic. Simple and neat, thanks! +1

; ;