Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive is cannot run query or cannot run hive in cli mode

Highlighted

Hive is cannot run query or cannot run hive in cli mode

New Contributor

We have some problem in our HDP envrionment (HDP 2.6.5)

1. We can access hive from ambari but when we query such as show tables it returns

Message:

 E090 HDFS020 Could not write file /user/admin/hive/jobs/hive-job-2060-2019-05-22_08-47/query.hql [HdfsApiException]

Trace:

 org.apache.ambari.view.utils.hdfs.HdfsApiException: HDFS020 Could not write file /user/admin/hive/jobs/hive-job-2061-2019-05-22_08-58/query.hql 
org.apache.ambari.view.utils.hdfs.HdfsApiException: HDFS020 Could not write file /user/admin/hive/jobs/hive-job-2061-2019-05-22_08-58/query.hql     at org.apache.ambari.view.utils.hdfs.HdfsUtil.putStringToFile(HdfsUtil.java:57)     at org.apache.ambari.view.hive20.resources.jobs.viewJobs.JobControllerImpl.setupQueryFile(JobControllerImpl.java:267)     at org....          (more...)


2. When run hive via cli it shows

log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.

Logging initialized using configuration in file:/etc/hive/2.6.5.0-292/0/hive-log4j.properties

>>>Stuck for long time in this section, no further section appear

3. Then when we try debug hive it returns

log4j:ERROR Could not find value for key log4j.appender.DEBUG
log4j:ERROR Could not instantiate appender named "DEBUG".
log4j:WARN No such property [maxFileSize] in org.apache.log4j.DailyRollingFileAppender.

Logging initialized using configuration in file:/etc/hive/2.6.5.0-292/0/hive-log4j.properties
19/05/22 09:04:08 [main]: INFO SessionState:
Logging initialized using configuration in file:/etc/hive/2.6.5.0-292/0/hive-log4j.properties
19/05/22 09:04:08 [main]: INFO hive.metastore: Trying to connect to metastore with URI thrift://mghdop02.dcdms:9083
19/05/22 09:04:08 [main]: INFO hive.metastore: Connected to metastore.
19/05/22 09:04:10 [main]: INFO session.SessionState: Created local directory: /tmp/7340c5fc-15bf-479c-8931-76b45c89900e_resources
19/05/22 09:04:10 [main]: INFO session.SessionState: Created HDFS directory: /tmp/hive/root/7340c5fc-15bf-479c-8931-76b45c89900e
19/05/22 09:04:10 [main]: INFO session.SessionState: Created local directory: /tmp/root/7340c5fc-15bf-479c-8931-76b45c89900e
19/05/22 09:04:10 [main]: INFO session.SessionState: Created HDFS directory: /tmp/hive/root/7340c5fc-15bf-479c-8931-76b45c89900e/_tmp_space.db
19/05/22 09:04:10 [main]: INFO tez.TezSessionState: Opening the session with id 7340c5fc-15bf-479c-8931-76b45c89900e for thread main log trace id - query id -
19/05/22 09:04:10 [main]: INFO tez.TezSessionState: User of session id 7340c5fc-15bf-479c-8931-76b45c89900e is root
19/05/22 09:04:10 [main]: INFO tez.DagUtils: Localizing resource because it does not exist: file:/usr/hdp/current/hive-webhcat/share/hcatalog/hive-hcatalog-core.jar to dest: hdfs://mghdop01.dcdms:8020/tmp/hive/root/_tez_session_dir/7340c5fc-15bf-479c-8931-76b45c89900e/hive-hcatalog-core.jar
19/05/22 09:04:11 [main]: INFO tez.DagUtils: Resource modification time: 1558490651312
19/05/22 09:04:11 [main]: INFO tez.DagUtils: Jar dir is null/directory doesn't exist. Choosing HIVE_INSTALL_DIR - /user/root/.hiveJars
19/05/22 09:04:11 [main]: INFO tez.DagUtils: Resource modification time: 1547439722103
19/05/22 09:04:11 [main]: INFO sqlstd.SQLStdHiveAccessController: Created SQLStdHiveAccessController for session context : HiveAuthzSessionContext [sessionString=7340c5fc-15bf-479c-8931-76b45c89900e, clientType=HIVECLI]
19/05/22 09:04:11 [main]: INFO hive.metastore: Mestastore configuration hive.metastore.filter.hook changed from org.apache.hadoop.hive.metastore.DefaultMetaStoreFilterHookImpl to org.apache.hadoop.hive.ql.security.authorization.plugin.AuthorizationMetaStoreFilterHook
19/05/22 09:04:11 [main]: INFO hive.metastore: Trying to connect to metastore with URI thrift://mghdop02.dcdms:9083
19/05/22 09:04:11 [main]: INFO hive.metastore: Connected to metastore.
19/05/22 09:04:12 [main]: INFO client.TezClient: Tez Client Version: [ component=tez-api, version=0.7.0.2.6.5.0-292, revision=2fcfb8aa6a462569e160756dfb73514f99cbf248, SCM-URL=scm:git:https://git-wip-us.apache.org/repos/asf/tez.git, buildTime=2018-05-11T07:53:39Z ]
19/05/22 09:04:12 [main]: INFO tez.TezSessionState: Opening new Tez Session (id: 7340c5fc-15bf-479c-8931-76b45c89900e, scratch dir: hdfs://mghdop01.dcdms:8020/tmp/hive/root/_tez_session_dir/7340c5fc-15bf-479c-8931-76b45c89900e)
19/05/22 09:04:12 [main]: INFO client.RMProxy: Connecting to ResourceManager at mghdop01.dcdms/10.0.37.157:8050
19/05/22 09:04:12 [main]: INFO client.AHSProxy: Connecting to Application History server at mghdop01.dcdms/10.0.37.157:10200
19/05/22 09:04:12 [main]: INFO client.TezClient: Session mode. Starting session.
19/05/22 09:04:12 [main]: INFO client.TezClientUtils: Using tez.lib.uris value from configuration: /hdp/apps/2.6.5.0-292/tez/tez.tar.gz
19/05/22 09:04:12 [main]: INFO client.TezClient: Tez system stage directory hdfs://mghdop01.dcdms:8020/tmp/hive/root/_tez_session_dir/7340c5fc-15bf-479c-8931-76b45c89900e/.tez/application_1558444524634_0004 doesn't exist and is created
19/05/22 09:04:12 [main]: INFO impl.YarnClientImpl: Submitted application application_1558444524634_0004
19/05/22 09:04:12 [main]: INFO client.TezClient: The url to track the Tez Session: http://mghdop01.dcdms:8088/proxy/application_1558444524634_0004/
19/05/22 09:05:13 [main]: INFO client.TezClient: App did not succeed. Diagnostics: Application application_1558444524634_0004 failed 2 times due to Error launching appattempt_1558444524634_0004_000002. Got exception: org.apache.hadoop.security.token.SecretManager$InvalidToken: Given NMToken for application : appattempt_1558444524634_0004_000002 is not valid for current node manager.expected : dnhdop02.dcdms:45454 found : dnhdop03.dcdms:45454
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:99)
at sun.reflect.GeneratedMethodAccessor248.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:290)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:202)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:184)
at com.sun.proxy.$Proxy91.startContainers(Unknown Source)
at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:125)
at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Given NMToken for application : appattempt_1558444524634_0004_000002 is not valid for current node manager.expected : dnhdop02.dcdms:45454 found : dnhdop03.dcdms:45454
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)
at org.apache.hadoop.ipc.Client.call(Client.java:1498)
at org.apache.hadoop.ipc.Client.call(Client.java:1398)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy90.startContainers(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
... 12 more
. Failing the application.
Exception in thread "main" java.lang.RuntimeException: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1558444524634_0004 failed 2 times due to Error launching appattempt_1558444524634_0004_000002. Got exception: org.apache.hadoop.security.token.SecretManager$InvalidToken: Given NMToken for application : appattempt_1558444524634_0004_000002 is not valid for current node manager.expected : dnhdop02.dcdms:45454 found : dnhdop03.dcdms:45454
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:99)
at sun.reflect.GeneratedMethodAccessor248.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:290)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:202)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:184)
at com.sun.proxy.$Proxy91.startContainers(Unknown Source)
at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:125)
at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Given NMToken for application : appattempt_1558444524634_0004_000002 is not valid for current node manager.expected : dnhdop02.dcdms:45454 found : dnhdop03.dcdms:45454
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)
at org.apache.hadoop.ipc.Client.call(Client.java:1498)
at org.apache.hadoop.ipc.Client.call(Client.java:1398)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy90.startContainers(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
... 12 more
. Failing the application.
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:233)
at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1558444524634_0004 failed 2 times due to Error launching appattempt_1558444524634_0004_000002. Got exception: org.apache.hadoop.security.token.SecretManager$InvalidToken: Given NMToken for application : appattempt_1558444524634_0004_000002 is not valid for current node manager.expected : dnhdop02.dcdms:45454 found : dnhdop03.dcdms:45454
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:99)
at sun.reflect.GeneratedMethodAccessor248.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:290)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:202)
at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:184)
at com.sun.proxy.$Proxy91.startContainers(Unknown Source)
at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:125)
at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:267)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken): Given NMToken for application : appattempt_1558444524634_0004_000002 is not valid for current node manager.expected : dnhdop02.dcdms:45454 found : dnhdop03.dcdms:45454
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)
at org.apache.hadoop.ipc.Client.call(Client.java:1498)
at org.apache.hadoop.ipc.Client.call(Client.java:1398)
at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
at com.sun.proxy.$Proxy90.startContainers(Unknown Source)
at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
... 12 more
. Failing the application.
at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:699)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:218)
at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:116)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:579)
... 8 more

What happen it our HDP and how to resolve this?
We have restart all service in ambari but unfortunate there aren't any effect