Error: Could not find or load main class org.apache.tez.runtime.task.TezChild

[#1]

Containers are being killed with error:

Container: container_e81_1447433228005_0589_01_001207 on datanode1.company.com_45454
=======================================================================================
LogType:stderr
Log Upload Time:Fri Nov 13 13:14:59 -0500 2015
LogLength:78
Log Contents:
Error: Could not find or load main class org.apache.tez.runtime.task.TezChild
--
Container: container_e81_1447433228005_0589_01_003434 on datanode2.company.com_45454
=======================================================================================
LogType:stderr
Log Upload Time:Fri Nov 13 13:14:58 -0500 2015
LogLength:78
Log Contents:
Error: Could not find or load main class org.apache.tez.runtime.task.TezChild
--
Container: container_e81_1447433228005_0589_01_003219 on datanode2.company.com_45454
=======================================================================================
LogType:stderr
Log Upload Time:Fri Nov 13 13:14:58 -0500 2015
LogLength:78
Log Contents:
Error: Could not find or load main class org.apache.tez.runtime.task.TezChild

Any ideas?

[#2]

@William Gonzalez

Please remove customer name from the logs

[#3]

@William Gonzalez

HDP version ?

[#4]

Thanks for the reminder

v2.2.4.2

[#5]

Can you check the installation of Tez? Detailed manual instructions are available here. Typically you see this when tez.lib.uris is not configured correctly.

[#6]

Thanks for the reply, Deepesh.

This cluster was configured by ambari and the Tez settings are correct.

Specifically, the page you cited talks about tez.lib.uris which is configured correctly:

<property> <name>tez.lib.uris</name> <value>/hdp/apps/2.2.4.5-6/tez/tez.tar.gz</value> </property>

All the other properties stated in that page also look correct to me.

Again, this is an intermittent issue. If we are missing properties, is not because of misconfiguration but rather the properties are being lost somehow.

Ideas?

[#7]

Suspect that the HDFS has some issues, can you look for exceptions in the DN and NN logs. Also scan for DFSClient exceptions in the the container logs.

[#8]

Thank you, Deepesh. I will do that.