Hi,
I have a oozie sqoop action. It works well with list-tables and list-databases commands. But it always fails with the import commands. Below is the error:
Log Type: stderr
Log Upload Time: Mon Oct 05 01:25:43 -0700 2020
Log Length: 2509
The system is out of resources.
Consult the following stack trace for details.
java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.sun.tools.javac.file.ZipFileIndex$ZipDirectory.readEntry(ZipFileIndex.java:669)
at com.sun.tools.javac.file.ZipFileIndex$ZipDirectory.buildIndex(ZipFileIndex.java:580)
at com.sun.tools.javac.file.ZipFileIndex$ZipDirectory.access$000(ZipFileIndex.java:484)
at com.sun.tools.javac.file.ZipFileIndex.checkIndex(ZipFileIndex.java:192)
at com.sun.tools.javac.file.ZipFileIndex.<init>(ZipFileIndex.java:137)
at com.sun.tools.javac.file.ZipFileIndexCache.getZipFileIndex(ZipFileIndexCache.java:100)
at com.sun.tools.javac.file.JavacFileManager.openArchive(JavacFileManager.java:548)
at com.sun.tools.javac.file.JavacFileManager.openArchive(JavacFileManager.java:482)
at com.sun.tools.javac.file.JavacFileManager.listContainer(JavacFileManager.java:368)
at com.sun.tools.javac.file.JavacFileManager.list(JavacFileManager.java:644)
at com.sun.tools.javac.jvm.ClassReader.fillIn(ClassReader.java:2499)
at com.sun.tools.javac.jvm.ClassReader.complete(ClassReader.java:2143)
at com.sun.tools.javac.code.Symbol.complete(Symbol.java:421)
at com.sun.tools.javac.comp.Enter.visitTopLevel(Enter.java:298)
at com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:459)
at com.sun.tools.javac.comp.Enter.classEnter(Enter.java:258)
at com.sun.tools.javac.comp.Enter.classEnter(Enter.java:272)
at com.sun.tools.javac.comp.Enter.complete(Enter.java:484)
at com.sun.tools.javac.comp.Enter.main(Enter.java:469)
at com.sun.tools.javac.main.JavaCompiler.enterTrees(JavaCompiler.java:929)
at com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:824)
at com.sun.tools.javac.main.Main.compile(Main.java:439)
at com.sun.tools.javac.api.JavacTaskImpl.call(JavacTaskImpl.java:132)
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:220)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
Intercepting System.exit(1)
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
Log Type: stdout
Log Upload Time: Mon Oct 05 01:25:43 -0700 2020
Log Length: 141911
Showing 4096 bytes of 141911 total. Click here for the full log.
.sqoop.manager.SqlManager - Executing SQL statement: SELECT t.* FROM `orders` AS t LIMIT 1
2020-10-05 01:25:26,955 [main] INFO org.apache.sqoop.manager.SqlManager - Executing SQL statement: SELECT t.* FROM `orders` AS t LIMIT 1
2020-10-05 01:25:26,961 [main] INFO org.apache.sqoop.orm.CompilationManager - HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
2020-10-05 01:25:35,437 [main] ERROR org.apache.sqoop.tool.ImportTool - Import failed: java.io.IOException: Error returned by javac
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:222)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:494)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:187)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:170)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:81)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:51)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:235)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:459)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
<<< Invocation of Sqoop command completed <<<
No child hadoop job is executed.
Intercepting System.exit(1)
<<< Invocation of Main class completed <<<
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
Oozie Launcher failed, finishing Hadoop job gracefully
Oozie Launcher, uploading action data to HDFS sequence file: hdfs://quickstart.cloudera:8020/user/cloudera/oozie-oozi/0000007-201004234744450-oozie-oozi-W/firstsqoopjob--sqoop/action-data.seq
2020-10-05 01:25:35,799 [main] INFO org.apache.hadoop.io.compress.zlib.ZlibFactory - Successfully loaded & initialized native-zlib library
2020-10-05 01:25:35,801 [main] INFO org.apache.hadoop.io.compress.CodecPool - Got brand-new compressor [.deflate]
Successfully reset security manager from org.apache.oozie.action.hadoop.LauncherSecurityManager@26a6a2f3 to null
Oozie Launcher ends
2020-10-05 01:25:35,859 [main] INFO org.apache.hadoop.mapred.Task - Task:attempt_1601880396378_0009_m_000000_0 is done. And is in the process of committing
2020-10-05 01:25:35,859 [main] INFO org.apache.hadoop.mapred.Task - Task:attempt_1601880396378_0009_m_000000_0 is done. And is in the process of committing
2020-10-05 01:25:35,997 [main] INFO org.apache.hadoop.mapred.Task - Task 'attempt_1601880396378_0009_m_000000_0' done.
2020-10-05 01:25:35,997 [main] INFO org.apache.hadoop.mapred.Task - Task 'attempt_1601880396378_0009_m_000000_0' done.
2020-10-05 01:25:35,998 [main] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - Stopping MapTask metrics system...
2020-10-05 01:25:36,000 [main] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - MapTask metrics system stopped.
2020-10-05 01:25:36,000 [main] INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl - MapTask metrics system shutdown complete.
workflow.xml
<workflow-app name='example-forkjoinwf' xmlns="uri:oozie:workflow:0.1">
<start to='firstsqoopjob' />
<action name="firstsqoopjob">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>[job tracker info]</job-tracker>
<name-node>[namenode info]</name-node>
<prepare>
<delete path="[name node info]/user/cloudera/nit/sqoop/import/oozietest"/>
<mkdir path="[name node info]/user/cloudera/nit/test6"/>
</prepare>
<configuration>
<property>
<name>mapred.compress.map.output</name>
<value>true</value>
</property>
</configuration>
<command>sqoop import --connect jdbc:mysql://hostname:3306/retail_db --username usr_hdp --password passcloud --table orders --target-dir /user/cloudera/nit/sqoop/import/oozietest --as-textfile --delete-target-dir -m 1</command>
</sqoop>
<ok to="end" />
<error to="kill" />
</action>
<kill name="kill">
<message>Map/Reduce failed, error message[${wf:errorMessage()}]</message>
</kill>
<end name='end'/>
</workflow-app>
