Reply
Ha
Explorer
Posts: 15
Registered: ‎02-19-2018

sqoop import-all-tables not creating hive metadata

 

Hi Everyone,

 

 I am running the script below hoping that the hive tables will be created automatically or unless I misunderstood the purpose of the command is not doing it. It looks like is dumping all the data of the tables into flat files but there is no  metadata or hive tables created at all. If this the way suppose to work or am I missing something...

thanks..

 

sqoop import-all-tables \
--direct \
--connect jdbc:postgresql://localhost/testdb \
--username hive \
--password hive \
-- --schema hivetest \
--m1 \
--hive-import \
--hive-database edh \
--hive-overwrite \
--as-textfile \
--warehouse-dir datawarehouse
--outdir java_out

 

Expert Contributor
Posts: 100
Registered: ‎02-23-2018

Re: sqoop import-all-tables not creating hive metadata

have you any console logs?


Regards.
Ha
Explorer
Posts: 15
Registered: ‎02-19-2018

Re: sqoop import-all-tables not creating hive metadata

Hi,

Yes I got a fresh output file... Also I just realise out the 1800 tables 1342 tables contains data so not sure what is going on.

18/02/23 21:56:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.13.1

18/02/23 21:56:05 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.

18/02/23 21:56:05 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override

18/02/23 21:56:05 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.

18/02/23 21:56:05 INFO manager.SqlManager: Using default fetchSize of 1000

18/02/23 21:56:05 INFO manager.PostgresqlManager: We will use schema claim

18/02/23 21:56:07 INFO tool.CodeGenTool: Beginning code generation

18/02/23 21:56:07 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM "claim"."employmentdata" AS t LIMIT 1

18/02/23 21:56:09 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/employmentdata.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.

18/02/23 21:56:11 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/employmentdata.jar

18/02/23 21:56:11 INFO manager.DirectPostgresqlManager: Beginning psql fast path import

18/02/23 21:56:12 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM "claim"."employmentdata" AS t LIMIT 1

18/02/23 21:56:13 INFO manager.DirectPostgresqlManager: Copy command is COPY (SELECT "wageamount", "loadcommandid", "overtimerate", "createuserid", "createtime", "ssbenefitsamnt", "updateuserid", "lastworkeddate", "updatetime", "scndinjryfnd", "wageamountpostinjury", "numhoursworked", "id", "paidfull", "daysworkedweek", "lastyearincome", "hirestate", "employmentstatus", "hiredate", "publicid", "scndinjryfnddate", "classcodeid", "beanversion", "retired", "scndinjryfndamnt", "archivepartition", "payperiod", "occupation", "numdaysworked", "wagepaymentcont", "injurystarttime", "ssbenefits", "departmentcode", "ctl_ins_ts", "ctl_upd_ts", "wageamountestimate_ext", "totalhoursworkorweek_ext", "nominatinterimweeklyamount_ext", "isthisaninterimpayment_ext", "ctpemployer", "wcdiscontfringebenefits_ext", "ctpoccupation", "wcworkweek_ext", "ctpreturnworkorschool", "wageamounteffectivedate_ext", "indexedpreinjuryearning_ext", "employmentfinalindicator", "dateceasedwork_ext", "nominatinterimamount_ext", "ctpemployername", "ctplatestreturnwork", "ctpemploymentstatus", "totalhoursworkperweek_ext", "dateterminated_ext", "terminatedfromwork_ext", "preinjuryearning_ext" FROM "claim"."employmentdata" WHERE 1=1) TO STDOUT WITH DELIMITER E'\1' CSV ;

18/02/23 21:56:13 INFO manager.DirectPostgresqlManager: Performing import of table employmentdata from database iadpprod

18/02/23 21:56:15 INFO manager.DirectPostgresqlManager: Transfer loop complete.

18/02/23 21:56:15 INFO manager.DirectPostgresqlManager: Transferred 150.1445 KB in 1.1874 seconds (126.4439 KB/sec)

18/02/23 21:56:16 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM "claim"."employmentdata" AS t LIMIT 1

18/02/23 21:56:18 WARN hive.TableDefWriter: Column wageamount had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column overtimerate had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column createtime had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column ssbenefitsamnt had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column lastworkeddate had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column updatetime had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column wageamountpostinjury had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column numhoursworked had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column lastyearincome had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column hiredate had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column scndinjryfnddate had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column scndinjryfndamnt had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column numdaysworked had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column injurystarttime had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column ctl_ins_ts had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column ctl_upd_ts had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column totalhoursworkorweek_ext had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column nominatinterimweeklyamount_ext had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column ctpemployer had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column wcdiscontfringebenefits_ext had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column ctpoccupation had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column wcworkweek_ext had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column ctpreturnworkorschool had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column wageamounteffectivedate_ext had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column indexedpreinjuryearning_ext had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column dateceasedwork_ext had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column ctplatestreturnwork had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column ctpemploymentstatus had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column totalhoursworkperweek_ext had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column dateterminated_ext had to be cast to a less precise type in Hive

18/02/23 21:56:18 WARN hive.TableDefWriter: Column preinjuryearning_ext had to be cast to a less precise type in Hive

18/02/23 21:56:18 INFO hive.HiveImport: Loading uploaded data into Hive



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 2.482 seconds

Loading data to table claim.employmentdata

Table claim.employmentdata stats: [numFiles=9, numRows=0, totalSize=1369008, rawDataSize=0]

OK

Time taken: 0.633 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/ctpmiscellaneous.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.041 seconds

Loading data to table claim.ctpmiscellaneous

Table claim.ctpmiscellaneous stats: [numFiles=8, numRows=0, totalSize=765589, rawDataSize=0]

OK

Time taken: 0.344 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/rdamacodes_ext.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.031 seconds

Loading data to table claim.rdamacodes_ext

Table claim.rdamacodes_ext stats: [numFiles=9, numRows=0, totalSize=543492007, rawDataSize=0]

OK

Time taken: 0.321 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/rdallctpinsurers.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.028 seconds

Loading data to table claim.rdallctpinsurers

Table claim.rdallctpinsurers stats: [numFiles=9, numRows=0, totalSize=90079, rawDataSize=0]

OK

Time taken: 0.309 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/sharingarrangement.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.03 seconds

Loading data to table claim.sharingarrangement

Table claim.sharingarrangement stats: [numFiles=9, numRows=0, totalSize=0, rawDataSize=0]

OK

Time taken: 0.307 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/rdstatulinecategory_ext.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.028 seconds

Loading data to table claim.rdstatulinecategory_ext

Table claim.rdstatulinecategory_ext stats: [numFiles=8, numRows=0, totalSize=527760, rawDataSize=0]

OK

Time taken: 0.305 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/incident.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.077 seconds

Loading data to table claim.incident

Table claim.incident stats: [numFiles=7, numRows=0, totalSize=6495961292, rawDataSize=0]

OK

Time taken: 0.395 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/ec_cr_request.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.028 seconds

Loading data to table claim.ec_cr_request

Table claim.ec_cr_request stats: [numFiles=3, totalSize=9754923]

OK

Time taken: 0.329 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/edge_ei_notify.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.026 seconds

Loading data to table claim.edge_ei_notify

Table claim.edge_ei_notify stats: [numFiles=3, totalSize=511395973]

OK

Time taken: 0.309 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/edge_ei_quantum.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.026 seconds

Loading data to table claim.edge_ei_quantum

Table claim.edge_ei_quantum stats: [numFiles=3, totalSize=511395196]

OK

Time taken: 0.304 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/edge_link_sritem.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.029 seconds

Loading data to table claim.edge_link_sritem

Table claim.edge_link_sritem stats: [numFiles=3, totalSize=24063]

OK

Time taken: 0.299 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/esm_relatedcontact.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.029 seconds

Loading data to table claim.esm_relatedcontact

Table claim.esm_relatedcontact stats: [numFiles=3, totalSize=107723056]

OK

Time taken: 0.314 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/esm_unavailability.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.03 seconds

Loading data to table claim.esm_unavailability

Table claim.esm_unavailability stats: [numFiles=3, totalSize=170810358]

OK

Time taken: 0.313 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/claimaccess.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.031 seconds

Loading data to table claim.claimaccess

Table claim.claimaccess stats: [numFiles=3, totalSize=2421535371]

OK

Time taken: 0.309 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/claiminfo.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.028 seconds

Loading data to table claim.claiminfo

Table claim.claiminfo stats: [numFiles=3, totalSize=603566927]

OK

Time taken: 0.292 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/claimmetricrecalctime.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.03 seconds

Loading data to table claim.claimmetricrecalctime

Table claim.claimmetricrecalctime stats: [numFiles=3, totalSize=437043868]

OK

Time taken: 0.286 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/excessgst.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.024 seconds

Loading data to table claim.excessgst

Table claim.excessgst stats: [numFiles=3, totalSize=172779306]

OK

Time taken: 0.282 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/fleet.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.024 seconds

Loading data to table claim.fleet

Table claim.fleet stats: [numFiles=3, totalSize=116224491]

OK

Time taken: 0.282 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/idrcustomerfeedback.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.032 seconds

Loading data to table claim.idrcustomerfeedback

Table claim.idrcustomerfeedback stats: [numFiles=4, numRows=0, totalSize=72822820, rawDataSize=0]

OK

Time taken: 0.317 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/idrhistorycomment.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.



Logging initialized using configuration in jar:file:/usr/lib/hive/lib/hive-common-1.1.0-cdh5.13.1.jar!/hive-log4j.properties

OK

Time taken: 0.028 seconds

Loading data to table claim.idrhistorycomment

Table claim.idrhistorycomment stats: [numFiles=3, totalSize=38475471]

OK

Time taken: 0.315 seconds

Note: /tmp/sqoop-cloud-user/compile/302583c717b75b956b23c8a549607d5d/inbound_notification.java uses or overrides a deprecated API.

Note: Recompile with -Xlint:deprecation for details.


Announcements