Support Questions
Find answers, ask questions, and share your expertise

Issue with Sqoop metastore

Hello All,

I have enabled Sqoop Metastore using MYSQL as the backend database using the instructions in this link. However, my oozie job is running into below error:

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], main() threw exception, sqoop-site.xml (Permission denied) java.io.FileNotFoundException: sqoop-site.xml (Permission denied) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:221) at java.io.FileOutputStream.<init>(FileOutputStream.java:110) at org.apache.oozie.action.hadoop.SqoopMain.setUpSqoopSite(SqoopMain.java:86) at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:160) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:51) at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:48) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:242) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

I have uploaded sqoop-site.xml in hdfs location - /user/oozie/share/lib/lib_20170119183643/sqoop/sqoop-site.xml and also restarted oozie server.

Find the oozie job log and sqoop-site.xml attached. Please help me to understand what location is it trying to look for sqoop-site.xml.

Thanks in advance.

Shashang Sheth

ooziejoblog.txt sqoop-site.xml

4 REPLIES 4

Re: Issue with Sqoop metastore

@Shashang Sheth

The issue might be due to Oozie tries to load the sqoop-site.xml to share/lib directory and fails due to permission for overwriting. Remove sqoop-site.xml from folder /user/oozie/share/lib/lib_20170119183643/sqoop and then try running the Oozie job.

Also, refer similar HCC link.

Re: Issue with Sqoop metastore

@Sindhu,

After removing sqoop-site.xml, the oozie job is now complaining on the contrary:

2017-02-15 12:51:05,420 WARN ActionStartXCommand:523 - SERVER[tlccmrhdd01.r1-core.r1.aig.net] USER[u1453711] GROUP[-] TOKEN[] APP[MDMRDM-RDM-tables] JOB[0000000-170215081728307-oozie-oozi-W] ACTION[0000000-170215081728307-oozie-oozi-W@Sqoop-exec-rdm-jobs] Error starting action [Sqoop-exec-rdm-jobs]. ErrorType [ERROR], ErrorCode [JA008], Message [JA008: File does not exist: hdfs://sandbox-gfat/user/oozie/share/lib/lib_20170119183643/sqoop/sqoop-site.xml#sqoop-site.xml] org.apache.oozie.action.ActionExecutorException: JA008: File does not exist: hdfs://sandbox-gfat/user/oozie/share/lib/lib_20170119183643/sqoop/sqoop-site.xml#sqoop-site.xml at org.apache.oozie.action.ActionExecutor.convertExceptionHelper(ActionExecutor.java:457) at org.apache.oozie.action.ActionExecutor.convertException(ActionExecutor.java:441) at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:1202) at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1373) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:232) at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63) at org.apache.oozie.command.XCommand.call(XCommand.java:287) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:331) at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:260) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:178) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744)

Re: Issue with Sqoop metastore

@Shashang Sheth

Please add it back and provide permissions for all users to read and execute, let me know.

After adding the sqoop-site.xml, make sure to run oozie admin-sharelibupdate to refresh the sharelib.

Re: Issue with Sqoop metastore

We decided to go with HSsqldb as our vendor wouldn't support sqoop metastore + mysql. Thanks, Sindhu for providing your inputs.