hive import fails from oozie

[#1]

I can run and execute a sqoop job from from CLI (with oozie user too) but when I am executing it from oozie workflow with following command:

why does it need a yarn user to execute it? I am running the job from oozie user only

	32066 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=WRITE, inode="/user/yarn":hdfs:hdfs:drwxr-xr-x
2016-05-14 17:45:20,615 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException: Permission denied: user=yarn, access=WRITE, inode="/user/yarn":hdfs:hdfs:drwxr-xr-x
32066 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
2016-05-14 17:45:20,615 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
32066 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
2016-05-14 17:45:20,615 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
32066 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)
2016-05-14 17:45:20,615 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)
32067 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
2016-05-14 17:45:20,616 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
32067 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1771)
2016-05-14 17:45:20,616 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1771)
32067 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1755)
2016-05-14 17:45:20,616 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1755)
32067 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1738)
2016-05-14 17:45:20,616 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1738)
32067 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71)
2016-05-14 17:45:20,616 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71)
32067 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3905)
2016-05-14 17:45:20,616 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3905)
32068 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1048)
2016-05-14 17:45:20,617 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1048)
32068 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
2016-05-14 17:45:20,617 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
32068 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
2016-05-14 17:45:20,617 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
32068 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
2016-05-14 17:45:20,617 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
32068 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
2016-05-14 17:45:20,617 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - 	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
32068 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2151)
2016-05-14 17:45:20,617 INFO  [Thread-30]

[#2]

It seems you're having a lot of issues with scheduling Oozie jobs. May I recommend you reach out to your local Hortonworks account representative and inquire about taking training courses. As a cost-effective solution, pick up a book or read a manual on Oozie and Sqoop webpages.

[#3]

You're right. Not that I have skipped the reading part but I would appreciate if you could suggest a fairly straightforward source. Also, I see the problem is with permissions here. Could you please check the update?@Artem Ervits

[#4 · Accepted solution]

@simran kaur

See this error

org.apache.sqoop.hive.HiveImport-Exceptionin thread "main" java.lang.RuntimeException: org.apache.hadoop.security.AccessControlException:Permission denied: user=yarn, access=WRITE, inode="/user/yarn":hdfs:hdfs:drwxr-xr-x

su - hdfs

hdfs dfs -chown -R yarn:hdfs /user/yarn

[#5]

@Neeraj Sabharwal

:Thanks. Just realized that yarn user is missing from hdfs. I can create the user but would like to know what are usually the reasons that sometimes users aren't created properly. It has happened before as well.

[#6]

Also, I did run service checks and it completed successfully.

[#7]

@simran kaur Ambari takes care of creating users. I am not sure how did you install the cluster. Did you use ambari?

[#8]

Yes ofcourse. I am using ambari and it did create rest of the users indeep but not yarn.I created the user and assigned it to the hdfs group. Ifact checked and found that the permissions there look like:

drwxrwxrwx - yarn hdfs 0 2016-05-14 19:14 /tmp/hive/yarn.

I am wondering why is it storing it in /tmp/hive/yarn ? It should store it in /tmp/hive/oozie when I am running it through oozie.NO?

I am still getting this: @Neeraj Sabharwal

- Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1463224371637_0088 failed 2 times due to AM Container for appattempt_1463224371637_0088_000002 exited with  exitCode: -1000
2016-05-14 19:45:56,425 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1463224371637_0088 failed 2 times due to AM Container for appattempt_1463224371637_0088_000002 exited with  exitCode: -1000
41399 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - For more detailed output, check application tracking page:http://warehouse.swtched.com:8088/cluster/app/application_1463224371637_0088Then, click on links to logs of each attempt.
2016-05-14 19:45:56,425 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - For more detailed output, check application tracking page:http://warehouse.swtched.com:8088/cluster/app/application_1463224371637_0088Then, click on links to logs of each attempt.
41399 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - Diagnostics: Permission denied: user=oozie, access=EXECUTE, inode="/tmp/hive/yarn/_tez_session_dir/f1b2db7d-0836-4330-849c-dc3e4d6dc2d1/hive-hcatalog-core.jar":yarn:hdfs:drwx------
2016-05-14 19:45:56,425 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - Diagnostics: Permission denied: user=oozie, access=EXECUTE, inode="/tmp/hive/yarn/_tez_session_dir/f1b2db7d-0836-4330-849c-dc3e4d6dc2d1/hive-hcatalog-core.jar":yarn:hdfs:drwx------
41399 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

[#9]

@simran kaur

try modifying the workflow.xml and add

<exec>${execFile}</exec> 

<env-var>HADOOP_USER_NAME=${wf:user()}</env-var> 
<file>${execPath}#${execFile}</file> 

[#10]

@Neeraj Sabharwal: But it also gives this in the error before permission denied:

I see this person had the same problem :https://community.hortonworks.com/questions/18261/tez-session-is-already-shutdown-failed-2-times-due.html . I would not really like to switch the engine to MR.

- Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1463224371637_0088 failed 2 times due to AM Container for appattempt_1463224371637_0088_000002 exited with  exitCode: -1000
2016-05-14 19:45:56,425 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1463224371637_0088 failed 2 times due to AM Container for appattempt_1463224371637_0088_000002 exited with  exitCode: -1000
41399 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - For more detailed output, check application tracking page:http://warehouse.swtched.com:8088/cluster/app/application_1463224371637_0088Then, click on links to logs of each attempt.
2016-05-14 19:45:56,425 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - For more detailed output, check application tracking page:http://warehouse.swtched.com:8088/cluster/app/application_1463224371637_0088Then, click on links to logs of each attempt.
41399 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - Diagnostics: Permission denied: user=oozie, access=EXECUTE, inode="/tmp/hive/yarn/_tez_session_dir/f1b2db7d-0836-4330-849c-dc3e4d6dc2d1/hive-hcatalog-core.jar":yarn:hdfs:drwx------
2016-05-14 19:45:56,425 INFO  [Thread-30] hive.HiveImport (LoggingAsyncSink.java:run(85)) - Diagnostics: Permission denied: user=oozie, access=EXECUTE, inode="/tmp/hive/yarn/_tez_session_dir/f1b2db7d-0836-4330-849c-dc3e4d6dc2d1/hive-hcatalog-core.jar":yarn:hdfs:drwx------
41399 [Thread-30] INFO  org.apache.sqoop.hive.HiveImport  - 	at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)

I have checked and TEZ is running. What could be the problem here?

checked the yarn logs applicationId id and got this:

ERROR org.apache.sqoop.tool.ImportTool  - Encountered IOException running import job: java.io.IOException: Hive exited with status 1
at org.apache.sqoop.hive.HiveImport.executeExternalHiveScript(HiveImport.java:394)
at org.apache.sqoop.hive.HiveImport.executeScript(HiveImport.java:344)
at org.apache.sqoop.hive.HiveImport.importTable(HiveImport.java:245)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:514)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
2016-05-14 19:45:56,806 ERROR [main] tool.ImportTool (ImportTool.java:run(613)) - Encountered IOException running import job: java.io.IOException: Hive exited with status 1