I can run and execute a sqoop job from from CLI (with oozie user too) but when I am executing it from oozie workflow with following command: why does it need a yarn user to execute it? I am running the job from oozie user only 32066 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=WRITE, inode="/user/yarn":hdfs:hdfs:drwxr-xr-x 2016-05-14 17:45:20,615 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=WRITE, inode="/user/yarn":hdfs:hdfs:drwxr-xr-x 32066 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) 2016-05-14 17:45:20,615 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319) 32066 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292) 2016-05-14 17:45:20,615 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292) 32066 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213) 2016-05-14 17:45:20,615 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213) 32067 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) 2016-05-14 17:45:20,616 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190) 32067 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1771) 2016-05-14 17:45:20,616 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1771) 32067 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1755) 2016-05-14 17:45:20,616 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1755) 32067 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1738) 2016-05-14 17:45:20,616 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1738) 32067 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71) 2016-05-14 17:45:20,616 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71) 32067 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3905) 2016-05-14 17:45:20,616 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3905) 32068 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1048) 2016-05-14 17:45:20,617 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1048) 32068 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) 2016-05-14 17:45:20,617 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622) 32068 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) 2016-05-14 17:45:20,617 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) 32068 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) 2016-05-14 17:45:20,617 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) 32068 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) 2016-05-14 17:45:20,617 INFO [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969) 32068 [Thread-30] INFO org.apache.sqoop.hive.HiveImport - at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151) 2016-05-14 17:45:20,617 INFO [Thread-30] ... View more

I have a coordinator running and have my first import successful but as soon as it runs for the second time, it throws file already exists exception. My question is how do I workaround this? 1. This isn't incremental import but a daily import of all data. 2. I do not want to delete it through <prepare> before I have a fresh copy of the import.(obviously because I would not want to lose what I have before I know I have a fresh and updated copy of the data) 3. I have --target-dir specified in the imports (Just in case that makes any diff. ) ... View more

I am executing an oozie job and get an error in oozie job/action description windows but nothing in the launcher logs Logs from oozie launcher: Log Type: stderr Log Upload Time: Wed May 11 15:48:00 +0530 2016 Log Length: 1721 May 11, 2016 3:47:44 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a provider class May 11, 2016 3:47:44 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as a provider class May 11, 2016 3:47:44 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices as a root resource class May 11, 2016 3:47:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate INFO: Initiating Jersey application, version 'Jersey: 1.9 09/02/2011 11:17 AM' May 11, 2016 3:47:44 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver to GuiceManagedComponentProvider with the scope "Singleton" May 11, 2016 3:47:44 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to GuiceManagedComponentProvider with the scope "Singleton" May 11, 2016 3:47:44 PM com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory getComponentProvider INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to GuiceManagedComponentProvider with the scope "PerRequest" log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Log Type: stdout Log Upload Time: Wed May 11 15:48:00 +0530 2016 Log Length: 0 but the windows shows oozie job failed with this error in oozie window: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] I am basically executing a sqoop job from oozie workflow. how do I check if there was an error while executing sqoop job?There is nothing in /log/sqoop either. ... View more

I changed the sqoop meta-store to a MySQL database and meta-connect is working fine. Also, added the updated sqoop-site.xml with changes made to connection url and password and added it to use/oozie/share/lib/lib_timestamp/sqoop/sqoop-site.xml Also, created and saved the sqoop jobs and these saved jobs are being run from oozie workflow. Now, on running oozie workflow, I am getting an error that says, Why is it not able to find sqoop-site.xml? I though added sqoop-site.xml to user/oozie/share/lib/sqoop folder after updating it, it never was in there before either. What could be the reason and how do I fix this? Log Type: stderr Log Upload Time: Mon May 09 10:54:39 +0530 2016 Log Length: 1868 Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], main() threw exception, sqoop-site.xml (Permission denied) java.io.FileNotFoundException: sqoop-site.xml (Permission denied) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.<init>(FileOutputStream.java:213) at java.io.FileOutputStream.<init>(FileOutputStream.java:101) at org.apache.oozie.action.hadoop.SqoopMain.setUpSqoopSite(SqoopMain.java:84) at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:155) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47) at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl). This is how I am running my sqoop jobs in oozie: <action name="fabricinventoryimport"> <sqoop xmlns="uri:oozie:sqoop-action:0.3"> <job-tracker>${jobTracker}</job-tracker> <name-node>${nameNode}</name-node> <command> job --meta-connect jdbc:mysql://FQDN:3306/sqoop --exec inventory </command> </sqoop> </action> ... View more

I hear that metastore in sqoop can take care of the incremental imports and that way I do not need to keep track of the last updated id/datetime myself. I am trying to execute this from an oozie WF but my question is 1,what goes into last-value parameter in sqoop command in that case(when I have a sqoop job and metastore configured)?(Do I need to even pass the parameter )? 2. Also, can I give multiple import statements in single sqoop job? 3. If yes, How? 4. Is it a good idea to execute multiple table imports in parallel? (I really would like to know the pros and cons attached to it). 5. If I plan to have table imports in parallel, do I just fork and execute jobs in oozie? ... View more