Support Questions
Find answers, ask questions, and share your expertise

Install HDP 2.6 using cloudbreak. Error on running HIVE on TEZ . Work in mapreduce

Explorer

Hi All

No problem on when i run Hive2 service check but when i run a query using tez it failed.

See below error

Logging initialized using configuration in file:/etc/hive/2.6.4.5-2/0/hive-log4j.properties Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1525674905503_0001 failed 2 times due to AM Container for appattempt_1525674905503_0001_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://ip-10-10-10-122.us-west-2.compute.internal:8088/cluster/app/application_1525674905503_0001 Then click on links to logs of each attempt. Diagnostics: ExitCodeException exitCode=2: tar: Removing leading `/' from member names tar: Skipping to next header tar: Exiting with failure status due to previous errors gzip: stdout: Broken pipe Failing this attempt. Failing the application. at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1525674905503_0001 failed 2 times due to AM Container for appattempt_1525674905503_0001_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://ip-10-10-10-122.us-west-2.compute.internal:8088/cluster/app/application_1525674905503_0001 Then click on links to logs of each attempt. Diagnostics: ExitCodeException exitCode=2: tar: Removing leading `/' from member names tar: Skipping to next header tar: Exiting with failure status due to previous errors gzip: stdout: Broken pipe Failing this attempt. Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:699) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:218) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:579)

gzip: stdout: Broken pipe Failing this attempt. Failing the application. at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1525674905503_0001 failed 2 times due to AM Container for appattempt_1525674905503_0001_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://ip-10-10-10-122.us-west-2.compute.internal:8088/cluster/app/application_1525674905503_0001 Then click on links to logs of each attempt. Diagnostics: ExitCodeException exitCode=2: tar: Removing leading `/' from member names tar: Skipping to next header tar: Exiting with failure status due to previous errors

Thanks in Advance

3 REPLIES 3

Re: Install HDP 2.6 using cloudbreak. Error on running HIVE on TEZ . Work in mapreduce

Explorer

Tez Service CHeck also failed

  File "/var/lib/ambari-agent/cache/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py", line 110, in <module>
    TezServiceCheck().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py", line 87, in service_check
    bin_dir = params.hadoop_bin_dir
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/execute_hadoop.py", line 44, in action_run
    environment = self.resource.environment,
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'hadoop --config /usr/hdp/2.6.4.5-2/hadoop/conf jar /usr/hdp/current/tez-client/tez-examples*.jar orderedwordcount /tmp/tezsmokeinput/sample-tez-test /tmp/tezsmokeoutput/' returned 255. 18/05/07 07:16:32 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.7.0.2.6.4.5-2, revision=0daef967e122a98f83b824f3b14991e792f5dd4d, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=2018-02-22T15:29:28Z ]
18/05/07 07:16:32 INFO client.RMProxy: Connecting to ResourceManager at ip-10-10-10-122.us-west-2.compute.internal/10.10.10.122:8050
18/05/07 07:16:32 INFO client.AHSProxy: Connecting to Application History server at ip-10-10-10-122.us-west-2.compute.internal/10.10.10.122:10200
18/05/07 07:16:33 INFO examples.OrderedWordCount: Running OrderedWordCount
18/05/07 07:16:33 INFO client.TezClient: Submitting DAG application with id: application_1525674905503_0004
18/05/07 07:16:33 INFO client.TezClientUtils: Using tez.lib.uris value from configuration: /hdp/apps/2.6.4.5-2/tez/tez.tar.gz
18/05/07 07:16:33 INFO client.TezClient: Tez system stage directory hdfs://ip-10-10-10-122.us-west-2.compute.internal:8020/tmp/ambari-qa/staging/.tez/application_1525674905503_0004 doesn't exist and is created
18/05/07 07:16:34 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1525674905503_0004, dagName=OrderedWordCount, callerContext={ context=TezExamples, callerType=null, callerId=null }
18/05/07 07:16:34 INFO impl.YarnClientImpl: Submitted application application_1525674905503_0004
18/05/07 07:16:34 INFO client.TezClient: The url to track the Tez AM: http://ip-10-10-10-122.us-west-2.compute.internal:8088/proxy/application_1525674905503_0004/
18/05/07 07:16:35 INFO client.TezClient: App did not succeed. Diagnostics: Application application_1525674905503_0004 failed 2 times due to AM Container for appattempt_1525674905503_0004_000002 exited with  exitCode: -1000
For more detailed output, check the application tracking page: http://ip-10-10-10-122.us-west-2.compute.internal:8088/cluster/app/application_1525674905503_0004 Then click on links to logs of each attempt.
Diagnostics: ExitCodeException exitCode=2: tar: Removing leading `/' from member names
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

gzip: stdout: Broken pipe

Failing this attempt. Failing the application.
18/05/07 07:16:35 INFO client.RMProxy: Connecting to ResourceManager at ip-10-10-10-122.us-west-2.compute.internal/10.10.10.122:8050
18/05/07 07:16:35 INFO client.AHSProxy: Connecting to Application History server at ip-10-10-10-122.us-west-2.compute.internal/10.10.10.122:10200
18/05/07 07:16:35 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
18/05/07 07:16:35 INFO examples.OrderedWordCount: DAG diagnostics: [Application application_1525674905503_0004 failed 2 times due to AM Container for appattempt_1525674905503_0004_000002 exited with  exitCode: -1000
For more detailed output, check the application tracking page: http://ip-10-10-10-122.us-west-2.compute.internal:8088/cluster/app/application_1525674905503_0004 Then click on links to logs of each attempt.
Diagnostics: ExitCodeException exitCode=2: tar: Removing leading `/' from member names
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

Re: Install HDP 2.6 using cloudbreak. Error on running HIVE on TEZ . Work in mapreduce

Expert Contributor

This is caused by a concurrency issue in Ambari and the tez.tar.gz gets corrupted during the copy to HDFS: https://issues.apache.org/jira/browse/AMBARI-23766

This issue is going to be fixed in Ambari 2.6.2, but as a workaround, please upload the tez.tar.gz manually:

#Switch to hdfs user
sudo su - hdfs

#Remove and upload the tez.tar.gz again
hdfs dfs -rm /hdp/apps/2.6.4.5-2/tez/tez.tar.gz
hdfs dfs -copyFromLocal /var/lib/ambari-agent/tmp/tez-native-tarball-staging/tez-native.tar.gz /hdp/apps/2.6.4.5-2/tez/tez.tar.gz
hdfs dfs -chown hdfs:hadoop /hdp/apps/2.6.4.5-2/tez/tez.tar.gz

I think you might need to restart HiveServer2, what you can do on Ambari UI.

Re: Install HDP 2.6 using cloudbreak. Error on running HIVE on TEZ . Work in mapreduce

Explorer

Thank you so much . It works