Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

I want to import certain tables from multiple SQL sever databases using sqoop to HDFS. Can someone guide me how to do it? an automated script would do well.

Solved Go to solution
Highlighted

I want to import certain tables from multiple SQL sever databases using sqoop to HDFS. Can someone guide me how to do it? an automated script would do well.

Contributor
 
1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: I want to import certain tables from multiple SQL sever databases using sqoop to HDFS. Can someone guide me how to do it? an automated script would do well.

Hi @omkar pathallapalli, a single Sqoop command can import only a single table from a given DB server. So, to import multiple tables from multiple servers you need a command, for example a Bash script like this

for tbl in $(cat $4); do
sqoop import –-connect "jdbc:sqlserver://${1}:3464;databaseName=${2}" --username ${3} -P --table ${tbl} --target-dir sqimport
done

And call it once per DB server providing each DB server FQDN, database name, user-name and a file listing the tables you want to import (one table per line). The script will prompt you for the password. You can modify target-dir and/or add more Sqoop properties including the number of mappers used for import, by default 4.

View solution in original post

3 REPLIES 3
Highlighted

Re: I want to import certain tables from multiple SQL sever databases using sqoop to HDFS. Can someone guide me how to do it? an automated script would do well.

Super Guru
Highlighted

Re: I want to import certain tables from multiple SQL sever databases using sqoop to HDFS. Can someone guide me how to do it? an automated script would do well.

Hi @omkar pathallapalli, a single Sqoop command can import only a single table from a given DB server. So, to import multiple tables from multiple servers you need a command, for example a Bash script like this

for tbl in $(cat $4); do
sqoop import –-connect "jdbc:sqlserver://${1}:3464;databaseName=${2}" --username ${3} -P --table ${tbl} --target-dir sqimport
done

And call it once per DB server providing each DB server FQDN, database name, user-name and a file listing the tables you want to import (one table per line). The script will prompt you for the password. You can modify target-dir and/or add more Sqoop properties including the number of mappers used for import, by default 4.

View solution in original post

Highlighted

Re: I want to import certain tables from multiple SQL sever databases using sqoop to HDFS. Can someone guide me how to do it? an automated script would do well.

Contributor

Nice solution @Predrag Minovic. Simple and neat, thanks! +1

Don't have an account?
Coming from Hortonworks? Activate your account here