Support Questions
Find answers, ask questions, and share your expertise

Getting error in sqoop import from Mysql into hive as Parquet Format

Getting error in sqoop import from Mysql into hive as Parquet Format

Explorer

Getting error in sqoop import from Mysql into hive as Parquet Format. But table is created but no data's in it.

Command:

sqoop import --connect jdbc:mysql://sandbox.hortonworks.com:3306/retail_db --username=retail_dba -P --table departments --hive-import --hive-home="/apps/hive/warehouse" --create-hive-table --hive-table departments --hive-database retail_edw --fields-terminated-by '|' --lines-terminated-by '\n' --as-parquetfile -m 4

Error as Below

16/05/15 15:31:07 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 16/05/15 15:31:07 INFO tool.CodeGenTool: Beginning code generation 16/05/15 15:31:07 INFO tool.CodeGenTool: Will generate java class as codegen_departments SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/hdp/2.4.0.0-169/hadoop/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/hdp/2.4.0.0-169/zookeeper/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 16/05/15 15:31:08 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `departments` AS t LIMIT 1 16/05/15 15:31:08 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `departments` AS t LIMIT 1 16/05/15 15:31:08 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.4.0.0-169/hadoop-mapreduce Note: /tmp/sqoop-root/compile/f26881f08d02b67aebb72e231c44fa1a/codegen_departments.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 16/05/15 15:31:11 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/f26881f08d02b67aebb72e231c44fa1a/codegen_departments.jar 16/05/15 15:31:11 WARN manager.MySQLManager: It looks like you are importing from mysql. 16/05/15 15:31:11 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 16/05/15 15:31:11 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 16/05/15 15:31:11 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 16/05/15 15:31:11 INFO mapreduce.ImportJobBase: Beginning import of departments 16/05/15 15:31:12 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `departments` AS t LIMIT 1 16/05/15 15:31:15 INFO hive.metastore: Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083 16/05/15 15:31:15 INFO hive.metastore: Connected to metastore. 16/05/15 15:31:15 INFO hive.metastore: Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083 16/05/15 15:31:15 INFO hive.metastore: Connected to metastore. 16/05/15 15:31:15 INFO hive.HiveManagedMetadataProvider: Creating a managed Hive table named: departments 16/05/15 15:31:16 INFO hive.metastore: Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083 16/05/15 15:31:16 INFO hive.metastore: Connected to metastore. 16/05/15 15:31:20 INFO impl.TimelineClientImpl: Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 16/05/15 15:31:20 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/192.168.183.132:8050 16/05/15 15:31:21 INFO hive.metastore: Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083 16/05/15 15:31:21 INFO hive.metastore: Connected to metastore. 16/05/15 15:31:24 INFO db.DBInputFormat: Using read commited transaction isolation 16/05/15 15:31:24 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`department_id`), MAX(`department_id`) FROM `departments` 16/05/15 15:31:25 INFO mapreduce.JobSubmitter: number of splits:4 16/05/15 15:31:25 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1463318852105_0012 16/05/15 15:31:27 INFO impl.YarnClientImpl: Submitted application application_1463318852105_0012 16/05/15 15:31:27 INFO mapreduce.Job: The url to track the job: http://sandbox.hortonworks.com:8088/proxy/application_1463318852105_0012/ 16/05/15 15:31:27 INFO mapreduce.Job: Running job: job_1463318852105_0012 16/05/15 15:31:41 INFO mapreduce.Job: Job job_1463318852105_0012 running in uber mode : false 16/05/15 15:31:41 INFO mapreduce.Job: map 0% reduce 0% 16/05/15 15:31:41 INFO mapreduce.Job: Job job_1463318852105_0012 failed with state FAILED due to: Application application_1463318852105_0012 failed 2 times due to AM Container for appattempt_1463318852105_0012_000002 exited with exitCode: 1 For more detailed output, check application tracking page:http://sandbox.hortonworks.com:8088/cluster/app/application_1463318852105_0012Then, click on links to logs of each attempt. Diagnostics: Exception from container-launch. Container id: container_e28_1463318852105_0012_02_000001 Exit code: 1 Stack trace: ExitCodeException exitCode=1: at org.apache.hadoop.util.Shell.runCommand(Shell.java:576) at org.apache.hadoop.util.Shell.run(Shell.java:487) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Container exited with a non-zero exit code 1 Failing this attempt. Failing the application. 16/05/15 15:31:41 INFO mapreduce.Job: Counters: 0 16/05/15 15:31:41 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 16/05/15 15:31:41 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 22.2023 seconds (0 bytes/sec) 16/05/15 15:31:41 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 16/05/15 15:31:41 INFO mapreduce.ImportJobBase: Retrieved 0 records. 16/05/15 15:31:41 ERROR tool.ImportTool: Error during import: Import job failed!

5 REPLIES 5

Re: Getting error in sqoop import from Mysql into hive as Parquet Format

@VENKATESH M

Please check the tasks attempt logs for this sqoop job on RM UI, It should have more info.

Re: Getting error in sqoop import from Mysql into hive as Parquet Format

@VENKATESH M

The logs you have posted are not useful, you need to check the tasks attempts logs and look for sys log and err.

You also need to click on the " Click here for the full log." to see full logs.

Re: Getting error in sqoop import from Mysql into hive as Parquet Format

Explorer

Plz find the RM UI logs:

Log Type: directory.info

Log Upload Time: Sun May 15 16:14:57 +0000 2016

Log Length: 23827

Showing 4096 bytes of 23827 total. Click here for the full log.

-r-xr-xr-x   1 yarn     hadoop      33361 Feb 10 06:42 ./mr-framework/hadoop/include/hdfs.h
789714    8 -r-xr-xr-x   1 yarn     hadoop       6330 Feb 10 06:42 ./mr-framework/hadoop/include/Pipes.hh
789729    4 drwxr-xr-x   4 yarn     hadoop       4096 Feb 10 06:42 ./mr-framework/hadoop/share
790085    4 drwxr-xr-x   3 yarn     hadoop       4096 Feb 10 06:42 ./mr-framework/hadoop/share/doc
926452    4 drwxr-xr-x   6 yarn     hadoop       4096 Feb 10 06:42 ./mr-framework/hadoop/share/doc/hadoop
789730    4 drwxr-xr-x   9 yarn     hadoop       4096 Feb 10 06:42 ./mr-framework/hadoop/share/hadoop
926043    4 drwxr-xr-x   6 yarn     hadoop       4096 Feb 10 06:42 ./mr-framework/hadoop/share/hadoop/common
926399    4 drwxr-xr-x   5 yarn     hadoop       4096 Feb 10 06:42 ./mr-framework/hadoop/share/hadoop/mapreduce
926279    4 drwxr-xr-x   5 yarn     hadoop       4096 Feb 10 06:42 ./mr-framework/hadoop/share/hadoop/tools
789973    4 drwxr-xr-x   3 yarn     hadoop       4096 Feb 10 06:42 ./mr-framework/hadoop/share/hadoop/httpfs
926140    4 drwxr-xr-x   7 yarn     hadoop       4096 Feb 10 06:42 ./mr-framework/hadoop/share/hadoop/hdfs
789842    4 drwxr-xr-x   3 yarn     hadoop       4096 Feb 10 06:42 ./mr-framework/hadoop/share/hadoop/kms
789731    4 drwxr-xr-x   5 yarn     hadoop       4096 Feb 10 06:42 ./mr-framework/hadoop/share/hadoop/yarn
790086    4 -r-xr-xr-x   1 yarn     hadoop       1366 Feb 10 06:42 ./mr-framework/hadoop/README.txt
789695    4 drwxr-xr-x   2 yarn     hadoop       4096 Feb 10 06:42 ./mr-framework/hadoop/bin
789698    4 -r-xr-xr-x   1 yarn     hadoop        830 Feb 10 06:42 ./mr-framework/hadoop/bin/kill-name-node
789706   12 -r-xr-xr-x   1 yarn     hadoop       8782 Feb 10 06:42 ./mr-framework/hadoop/bin/hdfs.cmd
789702    8 -r-xr-xr-x   1 yarn     hadoop       6594 Feb 10 06:42 ./mr-framework/hadoop/bin/hadoop
789699  128 -r-xr-xr-x   1 yarn     hadoop     129158 Feb 10 06:42 ./mr-framework/hadoop/bin/test-container-executor
789700   12 -r-xr-xr-x   1 yarn     hadoop      12069 Feb 10 06:42 ./mr-framework/hadoop/bin/yarn.cmd
789697   12 -r-xr-xr-x   1 yarn     hadoop       8786 Feb 10 06:42 ./mr-framework/hadoop/bin/hadoop.cmd
789705   16 -r-xr-xr-x   1 yarn     hadoop      12329 Feb 10 06:42 ./mr-framework/hadoop/bin/hdfs
789703    8 -r-xr-xr-x   1 yarn     hadoop       6059 Feb 10 06:42 ./mr-framework/hadoop/bin/mapred
789708    4 -r-xr-xr-x   1 yarn     hadoop       1857 Feb 10 06:42 ./mr-framework/hadoop/bin/rcc
789704  112 -r-xr-xr-x   1 yarn     hadoop     114438 Feb 10 06:42 ./mr-framework/hadoop/bin/container-executor
789701    4 -r-xr-xr-x   1 yarn     hadoop        888 Feb 10 06:42 ./mr-framework/hadoop/bin/kill-secondary-name-node
789707    8 -r-xr-xr-x   1 yarn     hadoop       7465 Feb 10 06:42 ./mr-framework/hadoop/bin/mapred.cmd
789696   16 -r-xr-xr-x   1 yarn     hadoop      14452 Feb 10 06:42 ./mr-framework/hadoop/bin/yarn
789715   16 -r-xr-xr-x   1 yarn     hadoop      15429 Feb 10 06:42 ./mr-framework/hadoop/LICENSE.txt
1580343   60 -r-x------   1 yarn     hadoop      60530 May 15 16:14 ./commons-logging-1.1.1.jar
1580373  152 -r-x------   1 yarn     hadoop     152255 May 15 16:14 ./parquet-hadoop-1.4.1.jar
1580225   12 -r-x------   1 yarn     hadoop      10378 May 15 16:14 ./parquet-generator-1.4.1.jar
1580397   20 -r-x------   1 yarn     hadoop      19816 May 15 16:14 ./opencsv-2.3.jar
1580210  392 -r-x------   1 yarn     hadoop     400090 May 15 16:14 ./avro-1.7.5.jar
1317565    4 drwxr-xr-x   2 yarn     hadoop       4096 May 15 16:14 ./jobSubmitDir
1055507    4 -r-x------   1 yarn     hadoop        488 May 15 16:14 ./jobSubmitDir/job.split
1055504    4 -r-x------   1 yarn     hadoop         25 May 15 16:14 ./jobSubmitDir/job.splitmetainfo
1577395 1736 -r-x------   1 yarn     hadoop    1774917 May 15 16:14 ./kite-data-hive-1.0.0.jar
1055515    4 -rwx------   1 yarn     hadoop        707 May 15 16:14 ./default_container_executor.sh
1577378 1704 -r-x------   1 yarn     hadoop    1741973 May 15 16:14 ./kite-data-mapreduce-1.0.0.jar

Re: Getting error in sqoop import from Mysql into hive as Parquet Format

Super Guru

Can you post the description of the mysql table departments with fields/data types and an example row or two?

wonder if there's odd data in there.

Also does that user have permissions to read the data in MYSQL?

Permissions to write to the Hive table?

Re: Getting error in sqoop import from Mysql into hive as Parquet Format

is this issue fixed? We are running into the same issue. Not able to import as a parquet file. Map red job fails.