Reply
Ha
Explorer
Posts: 15
Registered: ‎02-19-2018

is there a limit in the number of tables when using sqoop import-all-tables

 

 

 Hi ,

 

I just wondering if there is a limit in the number of tables that you can import by using  the sqoop import-all-tables command... I've tried a few times and even though I got 1800 tables in my source system after the sqoop import-all-tables completes it has done only 180 tables + there is no error or sign of failure. 

 

The command I am runing is shown below: 

 

sqoop import-all-tables \
--direct \
--connect jdbc:postgresql:/localhost/test \
--username hive -P \
--hive-import \
--hive-database hive \
-- --schema claim \
--num-mappers 4 \
--split-by id \
--hive-overwrite

 Any ideas... thanks in advance for your feedback...

 

 

 

 

 

Posts: 1,664
Kudos: 325
Solutions: 262
Registered: ‎07-31-2013

Re: is there a limit in the number of tables when using sqoop import-all-tables

There is no limit applied in the code. However, Sqoop will only pull
metadata of JDBC type 'TABLE', see:
https://github.com/cloudera/sqoop/blob/cdh5.14.0-release/src/java/org/apache/sqoop/manager/SqlManage...
and
https://github.com/cloudera/sqoop/blob/cdh5.14.0-release/src/java/org/apache/sqoop/tool/ImportAllTab...

Are your 1620 remaining objects under the DB all of type TABLE?
Ha
Explorer
Posts: 15
Registered: ‎02-19-2018

Re: is there a limit in the number of tables when using sqoop import-all-tables

I am not sure whether because the source DB is postgres so I wonder if it is a bit trickier for the sqoop code to identified the type.... And as far as I can see all the remainder of the objects are type 'TABLE'...

Posts: 1,664
Kudos: 325
Solutions: 262
Registered: ‎07-31-2013

Re: is there a limit in the number of tables when using sqoop import-all-tables

I imagine if Sqoop's list-tables utility (per:
http://archive.cloudera.com/cdh5/cdh/5/sqoop/SqoopUserGuide.html#_literal_sqoop_list_tables_literal)
works, so would the import, as they share the same manner of listing.

Can you also confirm if all of the remaining tables are non-empty? Also,
have you inspected the output of your import-all-tables in --verbose mode?
Ha
Explorer
Posts: 15
Registered: ‎02-19-2018

Re: is there a limit in the number of tables when using sqoop import-all-tables

Hi Harsh,

Thanks for your help... It looks those are basically empty tables and the output as shown below says 0 bytes transferred.
18/02/23 15:35:06 INFO manager.DirectPostgresqlManager: Transfer loop complete.
18/02/23 15:35:06 INFO manager.DirectPostgresqlManager: Transferred 0 bytes in 0.6186 seconds (0 bytes/sec)
18/02/23 15:35:07 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM "claim"."externaldisputetype" AS t LIMIT 1
So there is no way to copy only the metadata of these empty tables?
Thanks again...
Posts: 1,664
Kudos: 325
Solutions: 262
Registered: ‎07-31-2013

Re: is there a limit in the number of tables when using sqoop import-all-tables

I'm not aware of a way to copy over the metadata via Sqoop. The Hive import
gets triggered iff there's data currently, not otherwise.
Ha
Explorer
Posts: 15
Registered: ‎02-19-2018

Re: is there a limit in the number of tables when using sqoop import-all-tables

 

Hi Harsh,

 

 I tried different approach which is importing 1 table at the time using the script below:

 

export SCHEMANAME=$1
export TABLENAME=$2
sqoop import \
--direct \
--connect jdbc:postgresql://localhost/test \
--username hive \
--hive-import \
--table $TABLENAME \
--password hive \
--hive-database hive \
-- --schema $SCHEMANAME \
--num-mappers 4 \
--split-by id \
--hive-overwrite \
--outdir java_out \
--verbose

and this one goes through each table:

 

for x in `cat file.lst`; 
do  echo ${x};
 ./sqoop.cmd hive $x; 
done 

and seems to be working so not sure what is happening with the import-all-tables command but is not working as design...

Ha
Explorer
Posts: 15
Registered: ‎02-19-2018

Re: is there a limit in the number of tables when using sqoop import-all-tables

 

 Hi Harsh,

 

 Now I can confirm that the approach importing table by table works.... I managed to copy 1804 out the 1820 tables. There were few errors found on a smal bunch but what I can see even the empty tables metadata was copied succefully. the only issue is that the process is very very slow.

 

Highlighted
Ha
Explorer
Posts: 15
Registered: ‎02-19-2018

Re: is there a limit in the number of tables when using sqoop import-all-tables

 

Hi Harsh,

 

I wonder as you seems to be a cloudera employee maybe you can pass this onto the support team for further investigations. I can assure you that the import-all-tables doesn't when the source database is a postgress database. I can't raise a tickect to support because we haven't got an enterprise subscrition yet.. Let me know if you can follow this up with support. Thanks...

Announcements