Error with Tez jobs

[#1]

Hi, I am suddenly having these errors with my tez jobs scheduled in oozie.

Logging initialized using configuration in file:/etc/hive/2.4.0.0-169/0/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No cluster diagnostics found.
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:507)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No cluster diagnostics found.
	at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:730)
	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:217)
	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:117)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:504)
	... 8 more
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info

[#2]

@Rajkumar Singh log attached.

[#3]

@Joshua Adeleke looks resources issue here, from memory side it looks good but no of core available is only 1.

2016-12-13 09:10:51,742 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:168960, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 2401 lastPreemptionHeartbeat: 2400

[#4]

Either the Tez application did not start - in which case you will not find any YARN application for this.

OR

Your Tez application started but did not receive any DAG to run and timed out. You will find the exact reason in the YARN application master log for this job.

OR

Your Tez application started and crashed unexpectedly. You will find the exact reason in the YARN application master log for this job.

[#5]

@bikas I have ransacked the YARN logs. Would you please tell me where i can find the exact reason why these jobs are failing?

[#6]

If the AM timed out, then in the AM log you will find "Session timed out".

If the AM crashed, you will find an exception in the AM log or some error in the AM stderr/stdout.

[#7]

Try below command and let me know if it works for you. It worked for me.

hdfs dfs -chmod -R /tmp/hive

,

Try below command and let me know if it worked.

hdfs dfs -chmod -R 777 /tmp/hive