Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Install HDP 2.6 using cloudbreak. Error on running HIVE on TEZ . Work in mapreduce

Install HDP 2.6 using cloudbreak. Error on running HIVE on TEZ . Work in mapreduce

New Contributor

Hi All

No problem on when i run Hive2 service check but when i run a query using tez it failed.

See below error

Logging initialized using configuration in file:/etc/hive/2.6.4.5-2/0/hive-log4j.properties Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1525674905503_0001 failed 2 times due to AM Container for appattempt_1525674905503_0001_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://ip-10-10-10-122.us-west-2.compute.internal:8088/cluster/app/application_1525674905503_0001 Then click on links to logs of each attempt. Diagnostics: ExitCodeException exitCode=2: tar: Removing leading `/' from member names tar: Skipping to next header tar: Exiting with failure status due to previous errors gzip: stdout: Broken pipe Failing this attempt. Failing the application. at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1525674905503_0001 failed 2 times due to AM Container for appattempt_1525674905503_0001_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://ip-10-10-10-122.us-west-2.compute.internal:8088/cluster/app/application_1525674905503_0001 Then click on links to logs of each attempt. Diagnostics: ExitCodeException exitCode=2: tar: Removing leading `/' from member names tar: Skipping to next header tar: Exiting with failure status due to previous errors gzip: stdout: Broken pipe Failing this attempt. Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:699) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:218) at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:579)

gzip: stdout: Broken pipe Failing this attempt. Failing the application. at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:233) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1525674905503_0001 failed 2 times due to AM Container for appattempt_1525674905503_0001_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://ip-10-10-10-122.us-west-2.compute.internal:8088/cluster/app/application_1525674905503_0001 Then click on links to logs of each attempt. Diagnostics: ExitCodeException exitCode=2: tar: Removing leading `/' from member names tar: Skipping to next header tar: Exiting with failure status due to previous errors

Thanks in Advance

3 REPLIES 3

Re: Install HDP 2.6 using cloudbreak. Error on running HIVE on TEZ . Work in mapreduce

New Contributor

Tez Service CHeck also failed

  File "/var/lib/ambari-agent/cache/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py", line 110, in <module>
    TezServiceCheck().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 375, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/common-services/TEZ/0.4.0.2.1/package/scripts/service_check.py", line 87, in service_check
    bin_dir = params.hadoop_bin_dir
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/providers/execute_hadoop.py", line 44, in action_run
    environment = self.resource.environment,
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/system.py", line 262, in action_run
    tries=self.resource.tries, try_sleep=self.resource.try_sleep)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 72, in inner
    result = function(command, **kwargs)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 102, in checked_call
    tries=tries, try_sleep=try_sleep, timeout_kill_strategy=timeout_kill_strategy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 150, in _call_wrapper
    result = _call(command, **kwargs_copy)
  File "/usr/lib/ambari-agent/lib/resource_management/core/shell.py", line 303, in _call
    raise ExecutionFailed(err_msg, code, out, err)
resource_management.core.exceptions.ExecutionFailed: Execution of 'hadoop --config /usr/hdp/2.6.4.5-2/hadoop/conf jar /usr/hdp/current/tez-client/tez-examples*.jar orderedwordcount /tmp/tezsmokeinput/sample-tez-test /tmp/tezsmokeoutput/' returned 255. 18/05/07 07:16:32 INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.7.0.2.6.4.5-2, revision=0daef967e122a98f83b824f3b14991e792f5dd4d, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=2018-02-22T15:29:28Z ]
18/05/07 07:16:32 INFO client.RMProxy: Connecting to ResourceManager at ip-10-10-10-122.us-west-2.compute.internal/10.10.10.122:8050
18/05/07 07:16:32 INFO client.AHSProxy: Connecting to Application History server at ip-10-10-10-122.us-west-2.compute.internal/10.10.10.122:10200
18/05/07 07:16:33 INFO examples.OrderedWordCount: Running OrderedWordCount
18/05/07 07:16:33 INFO client.TezClient: Submitting DAG application with id: application_1525674905503_0004
18/05/07 07:16:33 INFO client.TezClientUtils: Using tez.lib.uris value from configuration: /hdp/apps/2.6.4.5-2/tez/tez.tar.gz
18/05/07 07:16:33 INFO client.TezClient: Tez system stage directory hdfs://ip-10-10-10-122.us-west-2.compute.internal:8020/tmp/ambari-qa/staging/.tez/application_1525674905503_0004 doesn't exist and is created
18/05/07 07:16:34 INFO client.TezClient: Submitting DAG to YARN, applicationId=application_1525674905503_0004, dagName=OrderedWordCount, callerContext={ context=TezExamples, callerType=null, callerId=null }
18/05/07 07:16:34 INFO impl.YarnClientImpl: Submitted application application_1525674905503_0004
18/05/07 07:16:34 INFO client.TezClient: The url to track the Tez AM: http://ip-10-10-10-122.us-west-2.compute.internal:8088/proxy/application_1525674905503_0004/
18/05/07 07:16:35 INFO client.TezClient: App did not succeed. Diagnostics: Application application_1525674905503_0004 failed 2 times due to AM Container for appattempt_1525674905503_0004_000002 exited with  exitCode: -1000
For more detailed output, check the application tracking page: http://ip-10-10-10-122.us-west-2.compute.internal:8088/cluster/app/application_1525674905503_0004 Then click on links to logs of each attempt.
Diagnostics: ExitCodeException exitCode=2: tar: Removing leading `/' from member names
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

gzip: stdout: Broken pipe

Failing this attempt. Failing the application.
18/05/07 07:16:35 INFO client.RMProxy: Connecting to ResourceManager at ip-10-10-10-122.us-west-2.compute.internal/10.10.10.122:8050
18/05/07 07:16:35 INFO client.AHSProxy: Connecting to Application History server at ip-10-10-10-122.us-west-2.compute.internal/10.10.10.122:10200
18/05/07 07:16:35 INFO client.DAGClientImpl: DAG completed. FinalState=FAILED
18/05/07 07:16:35 INFO examples.OrderedWordCount: DAG diagnostics: [Application application_1525674905503_0004 failed 2 times due to AM Container for appattempt_1525674905503_0004_000002 exited with  exitCode: -1000
For more detailed output, check the application tracking page: http://ip-10-10-10-122.us-west-2.compute.internal:8088/cluster/app/application_1525674905503_0004 Then click on links to logs of each attempt.
Diagnostics: ExitCodeException exitCode=2: tar: Removing leading `/' from member names
tar: Skipping to next header
tar: Exiting with failure status due to previous errors

Re: Install HDP 2.6 using cloudbreak. Error on running HIVE on TEZ . Work in mapreduce

Expert Contributor

This is caused by a concurrency issue in Ambari and the tez.tar.gz gets corrupted during the copy to HDFS: https://issues.apache.org/jira/browse/AMBARI-23766

This issue is going to be fixed in Ambari 2.6.2, but as a workaround, please upload the tez.tar.gz manually:

#Switch to hdfs user
sudo su - hdfs

#Remove and upload the tez.tar.gz again
hdfs dfs -rm /hdp/apps/2.6.4.5-2/tez/tez.tar.gz
hdfs dfs -copyFromLocal /var/lib/ambari-agent/tmp/tez-native-tarball-staging/tez-native.tar.gz /hdp/apps/2.6.4.5-2/tez/tez.tar.gz
hdfs dfs -chown hdfs:hadoop /hdp/apps/2.6.4.5-2/tez/tez.tar.gz

I think you might need to restart HiveServer2, what you can do on Ambari UI.

Highlighted

Re: Install HDP 2.6 using cloudbreak. Error on running HIVE on TEZ . Work in mapreduce

New Contributor

Thank you so much . It works

Don't have an account?
Coming from Hortonworks? Activate your account here