Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Error with Tez jobs

avatar
Expert Contributor

Hi, I am suddenly having these errors with my tez jobs scheduled in oozie.

Logging initialized using configuration in file:/etc/hive/2.4.0.0-169/0/hive-log4j.properties
Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No cluster diagnostics found.
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:507)
	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:680)
	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:624)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. No cluster diagnostics found.
	at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:730)
	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:217)
	at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:117)
	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:504)
	... 8 more
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.ShellMain], exit code [1]
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.impl.MetricsSystemImpl).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info
15 REPLIES 15

avatar
Expert Contributor

@Rajkumar Singh log attached.

avatar
Super Guru

@Joshua Adeleke looks resources issue here, from memory side it looks good but no of core available is only 1.

2016-12-13 09:10:51,742 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:168960, vCores:1> pendingRequests: 0 delayedContainers: 0 heartbeats: 2401 lastPreemptionHeartbeat: 2400

avatar
Super Collaborator

Either the Tez application did not start - in which case you will not find any YARN application for this.

OR

Your Tez application started but did not receive any DAG to run and timed out. You will find the exact reason in the YARN application master log for this job.

OR

Your Tez application started and crashed unexpectedly. You will find the exact reason in the YARN application master log for this job.

avatar
Expert Contributor

@bikas I have ransacked the YARN logs. Would you please tell me where i can find the exact reason why these jobs are failing?

avatar
Super Collaborator

If the AM timed out, then in the AM log you will find "Session timed out".

If the AM crashed, you will find an exception in the AM log or some error in the AM stderr/stdout.

avatar
New Contributor

Try below command and let me know if it works for you. It worked for me.

hdfs dfs -chmod -R /tmp/hive

,

Try below command and let me know if it worked.

hdfs dfs -chmod -R 777 /tmp/hive