Support Questions

Find answers, ask questions, and share your expertise

Tez job is failing because the one of the jar modified at runtime.

New Contributor

We observed that hcatlog-core.jar is getting modified while the job is running. Here is the log message. I am not sure if the log message is misleading or telling the correct information?

2018-04-02 22:17:09,062 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1520978446018_457642_1][Event:CONTAINER_STOPPED]: containerId=container_e85_1520978446018_457642_01_000004, stoppedTime=1522707429062, exitStatus=-1000
2018-04-02 22:17:09,067 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1520978446018_457642_1][Event:TASK_ATTEMPT_FINISHED]: vertexName=Map 1, taskAttemptId=attempt_1520978446018_457642_1_00_000000_0, creationTime=1522707419007, allocationTime=1522707423731, startTime=0, finishTime=1522707429064, timeTaken=1522707429064, status=FAILED, errorEnum=CONTAINER_EXITED, diagnostics=Container container_e85_1520978446018_457642_01_000004 finished with diagnostics set to [Container failed, exitCode=-1000. Resource hdfs://hdfsadlproduction/tmp/hive/mrcsub_adl_prod/_tez_session_dir/3e2bd935-a9dd-4f73-97e8-e74aebe8a96b/hive-hcatalog-core.jar changed on src filesystem (expected 1522707341791, was 1522707393297
java.io.IOException: Resource hdfs://hdfsadlproduction/tmp/hive/mrcsub_adl_prod/_tez_session_dir/3e2bd935-a9dd-4f73-97e8-e74aebe8a96b/hive-hcatalog-core.jar changed on src filesystem (expected 1522707341791, was 1522707393297
        at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:255)
        at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
        at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:358)
        at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Any pointers to narrow the down the problem is appreciated.

2 REPLIES 2

New Contributor
@Anandha Ranganathan

which version of hive are you using?

New Contributor

@puja

Here are the version info.

HDP Version : HDP-2.5.0.55

Hive : 1.2.1000

Tez: 0.7.0

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.