Cloudera Community

Announcements
Celebrating as our community reaches 100,000 members! Thank you!
jass
user rank
Explorer

Unable to create a hive table from the .csv file t...

by Explorer in Support Questions
‎05-03-2022 12:26 AM
‎05-03-2022 12:26 AM
We specified the environment variables as below:   export AWS_ACCESS_KEY_ID=xMK6bdX8iY************************************** export AWS_SECRET_KEY=34***************************************   After connecting to a hive session , we specified the s3a credentials:   set fs.s3a.endpoint=cluster.domain.*; set fs.s3a.access.key=$$$$$$$$$$$$$$###; set fs.s3a.secret.key=####$$$$;   Tried to create a table using the below query (with directory location in s3 bucket : (s3a://test/dir2/) and received the preceding error ; even though the s3 credentials were already specified as stated above:   0: jdbc:hive2://> CREATE EXTERNAL TABLE s3dir ( . . . . . . . . > col1 int, . . . . . . . . > col2 string, . . . . . . . . > col3 string, . . . . . . . . > col4 string . . . . . . . . > ) . . . . . . . . > ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' . . . . . . . . > LOCATION 's3a://test/dir2/' . . . . . . . . > TBLPROPERTIES ( . . . . . . . . > "s3select.format" = "csv" . . . . . . . . > ); 22/05/03 03:06:32 [2199007f-0721-4e46-89b6-40cef824235c main]: WARN impl.MetricsConfig: Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties 22/05/03 03:06:36 [HiveServer2-Background-Pool: Thread-71]: ERROR exec.Task: Failed org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.nio.file.AccessDeniedException s3a://test/dir2: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))) at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1170) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1175) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.createTableNonReplaceMode(CreateTableOperation.java:140) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation.execute(CreateTableOperation.java:98) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:82) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:357) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:749) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:504) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.Driver.run(Driver.java:498) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:88) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:327) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_322] at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_322] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:345) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322] Caused by: org.apache.hadoop.hive.metastore.api.MetaException: Got exception: java.nio.file.AccessDeniedException s3a://test/dir2: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63918) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result$create_table_req_resultStandardScheme.read(ThriftHiveMetastore.java:63886) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$create_table_req_result.read(ThriftHiveMetastore.java:63812) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_create_table_req(ThriftHiveMetastore.java:1796) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.create_table_req(ThriftHiveMetastore.java:1783) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.create_table_with_environment_context(HiveMetaStoreClient.java:3622) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.create_table_with_environment_context(SessionHiveMetaStoreClient.java:145) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1082) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createTable(HiveMetaStoreClient.java:1067) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:213) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at com.sun.proxy.$Proxy35.createTable(Unknown Source) ~[?:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:3515) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at com.sun.proxy.$Proxy35.createTable(Unknown Source) ~[?:?] at org.apache.hadoop.hive.ql.metadata.Hive.createTable(Hive.java:1159) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 28 more 22/05/03 03:06:36 [HiveServer2-Background-Pool: Thread-71]: ERROR exec.Task: DDLTask failed, DDL Operation: class org.apache.hadoop.hive.ql.ddl.table.create.CreateTableOperation org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:Got exception: java.nio.file.AccessDeniedException s3a://test/dir2: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))) ERROR : FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. MetaException(message:Got exception: java.nio.file.AccessDeniedException s3a://test/dir2: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))) Error: Error while compiling statement: FAILED: Execution Error, return code 40000 from org.apache.hadoop.hive.ql.ddl.DDLTask. MetaException(message:Got exception: java.nio.file.AccessDeniedException s3a://test/dir2: org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider IAMInstanceCredentialsProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))) (state=08S01,code=40000)     however the same works when the .csv file is present directly in s3 bucket and not inside any directory ('s3a://test/):     0: jdbc:hive2://> CREATE EXTERNAL TABLE s3notdir ( . . . . . . . . > col1 int, . . . . . . . . > col2 string, . . . . . . . . > col3 string, . . . . . . . . > col4 string . . . . . . . . > ) . . . . . . . . > ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' . . . . . . . . > LOCATION 's3a://test/' . . . . . . . . > TBLPROPERTIES ( . . . . . . . . > "s3select.format" = "csv" . . . . . . . . > ); OK No rows affected (2.223 seconds) 0: jdbc:hive2://>     ... View more
Labels:

Re: Unable to Start Hiveserver2 and unable to rea...

by Explorer in Support Questions
‎04-29-2022 02:20 AM
‎04-29-2022 02:20 AM
Thank you for the steps. hiveserver2 is up and running, but facing the below still in the hive session: [root@b7-40 ~]# hive SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Connecting to jdbc:hive2://b7-33.lab.archivas.com:2181,b7-40.lab.archivas.com:2181,b7-6.lab.archivas.com:2181/default;password=root;serviceDiscoveryMode=zooKeeper;ssl=true;user=root;zooKeeperNamespace=hiveserver2 22/04/29 03:00:25 [main]: WARN jdbc.HiveConnection: Failed to connect to b7-40.lab.archivas.com:10000 22/04/29 03:00:25 [main]: ERROR jdbc.Utils: Unable to read HiveServer2 configs from ZooKeeper Unknown HS2 problem when communicating with Thrift server. Error: Could not open client transport for any of the Server URI's in ZooKeeper: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (state=08S01,code=0) Beeline version 3.1.3000.7.1.7.1000-141 by Apache Hive beeline> !connect jdbc:hive2://b7-40.lab.archivas.com:10000 Connecting to jdbc:hive2://b7-40.lab.archivas.com:10000 Enter username for jdbc:hive2://b7-40.lab.archivas.com:10000: root Enter password for jdbc:hive2://b7-40.lab.archivas.com:10000: ******** 22/04/29 03:01:22 [main]: WARN jdbc.HiveConnection: Failed to connect to b7-40.lab.archivas.com:10000 Unknown HS2 problem when communicating with Thrift server. Error: Could not open client transport with JDBC Uri: jdbc:hive2://b7-40.lab.archivas.com:10000: Invalid status 21 (state=08S01,code=0) beeline> !connect jdbc:hive2://b7-33.lab.archivas.com:10000 Connecting to jdbc:hive2://b7-33.lab.archivas.com:10000 Enter username for jdbc:hive2://b7-33.lab.archivas.com:10000: root Enter password for jdbc:hive2://b7-33.lab.archivas.com:10000: ******** 22/04/29 03:01:44 [main]: WARN jdbc.HiveConnection: Failed to connect to b7-33.lab.archivas.com:10000 Could not open connection to the HS2 server. Please check the server URI and if the URI is correct, then ask the administrator to check the server status. Error: Could not open client transport with JDBC Uri: jdbc:hive2://b7-33.lab.archivas.com:10000: java.net.ConnectException: Connection refused (Connection refused) (state=08S01,code=0) beeline> !connect jdbc:hive2://b7-40.lab.archivas.com:10002 Connecting to jdbc:hive2://b7-40.lab.archivas.com:10002 Enter username for jdbc:hive2://b7-40.lab.archivas.com:10002: root Enter password for jdbc:hive2://b7-40.lab.archivas.com:10002: ******** 22/04/29 03:02:29 [main]: WARN jdbc.HiveConnection: Failed to connect to b7-40.lab.archivas.com:10002 Unknown HS2 problem when communicating with Thrift server. Error: Could not open client transport with JDBC Uri: jdbc:hive2://b7-40.lab.archivas.com:10002: Invalid status 21 (state=08S01,code=0) ... View more

Hiveserver2 stops working after being up for about...

by Explorer in Support Questions
‎04-26-2022 10:34 PM
‎04-26-2022 10:34 PM
1:25:46.144 AM INFO TezClient [main]: Failed to retrieve AM Status via proxy com.google.protobuf.ServiceException: java.io.EOFException: End of File Exception between local host is: "b7-40.lab.archivas.com/172.18.105.90"; destination host is: "b7-35.lab.archivas.com":41485; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:247) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at com.sun.proxy.$Proxy44.getAMStatus(Unknown Source) ~[?:?] at org.apache.tez.client.FrameworkClient.getAMStatus(FrameworkClient.java:115) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:843) [tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:977) [tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) [tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: java.io.EOFException: End of File Exception between local host is: "b7-40.lab.archivas.com/172.18.105.90"; destination host is: "b7-35.lab.archivas.com":41485; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_322] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_322] at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:892) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:846) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1508) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1405) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] ... 27 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) ~[?:1.8.0_322] at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1880) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1191) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1087) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] 1:25:51.874 AM INFO TezClient [main]: Failed to retrieve AM Status via proxy com.google.protobuf.ServiceException: java.io.EOFException: End of File Exception between local host is: "b7-40.lab.archivas.com/172.18.105.90"; destination host is: "b7-37.lab.archivas.com":32988; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:247) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at com.sun.proxy.$Proxy44.getAMStatus(Unknown Source) ~[?:?] at org.apache.tez.client.FrameworkClient.getAMStatus(FrameworkClient.java:115) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.getAppMasterStatus(TezClient.java:843) [tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:977) [tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) [tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) [hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: java.io.EOFException: End of File Exception between local host is: "b7-40.lab.archivas.com/172.18.105.90"; destination host is: "b7-37.lab.archivas.com":32988; : java.io.EOFException; For more details see: http://wiki.apache.org/hadoop/EOFException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_322] at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_322] at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:892) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:846) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1566) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1508) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.Client.call(Client.java:1405) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] ... 27 more Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:392) ~[?:1.8.0_322] at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1880) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1191) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1087) ~[hadoop-common-3.1.1.7.1.7.1000-141.jar:?] 1:25:52.379 AM INFO TezClient [main]: App did not succeed. Diagnostics: Application application_1650968366405_0009 failed 2 times due to AM Container for appattempt_1650968366405_0009_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-27 01:25:51.885]Exception from container-launch. Container id: container_e02_1650968366405_0009_02_000001 Exit code: 1 [2022-04-27 01:25:51.890]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : [2022-04-27 01:25:51.892]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : For more detailed output, check the application tracking page: https://b7-40.lab.archivas.com:8090/cluster/app/application_1650968366405_0009 Then click on links to logs of each attempt. . Failing the application. 1:25:52.380 AM ERROR HiveServer2 [main]: Error starting Tez sessions: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650968366405_0009 failed 2 times due to AM Container for appattempt_1650968366405_0009_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-27 01:25:51.885]Exception from container-launch. Container id: container_e02_1650968366405_0009_02_000001 Exit code: 1 [2022-04-27 01:25:51.890]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : [2022-04-27 01:25:51.892]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : For more detailed output, check the application tracking page: https://b7-40.lab.archivas.com:8090/cluster/app/application_1650968366405_0009 Then click on links to logs of each attempt. . Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:979) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 12 more 1:25:52.382 AM INFO HiveServer2 [main]: Shutting down HiveServer2 1:25:52.382 AM INFO ThriftCLIService [main]: Thrift server has stopped 1:25:52.382 AM INFO AbstractService [main]: Service:ThriftBinaryCLIService is stopped. 1:25:52.382 AM INFO AbstractService [main]: Service:OperationManager is stopped. 1:25:52.382 AM INFO AbstractService [main]: Service:SessionManager is stopped. 1:25:52.384 AM INFO AbstractService [main]: Service:CLIService is stopped. 1:25:52.385 AM INFO HiveMetaStoreClient [main]: Closed a connection to metastore, current connections: 16 1:25:52.385 AM INFO AbstractService [main]: Service:HiveServer2 is stopped. 1:25:53.066 AM INFO ScheduledQueryExecutionService [Scheduled Query Poller]: Thread finished; renaming back to: Scheduled Query Thread 0 1:25:53.386 AM WARN ScheduledQueryExecutionService [Scheduled Query Progress Reporter]: interrupt discarded 1:25:53.386 AM INFO ScheduledQueryExecutionService [Scheduled Query Progress Reporter]: Thread finished; renaming back to: Scheduled Query Thread 1 1:25:53.388 AM INFO AbstractConnector [main]: Stopped ServerConnector@7df4709e{SSL, (ssl, http/1.1)}{0.0.0.0:10002} 1:25:53.389 AM INFO session [main]: node0 Stopped scavenging 1:25:53.390 AM INFO ContextHandler [main]: Stopped o.e.j.s.ServletContextHandler@7ab2ba47{static,/static,jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/hive-service-3.1.3000.7.1.7.1000-141.jar!/hive-webapps/static,STOPPED} 1:25:53.392 AM INFO ContextHandler [main]: Stopped o.e.j.w.WebAppContext@559fd5ec{hiveserver2,/,null,STOPPED}{jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/hive-service-3.1.3000.7.1.7.1000-141.jar!/hive-webapps/hiveserver2} 1:25:53.394 AM INFO HiveServer2 [main]: Web UI has stopped 1:25:53.401 AM WARN ZooKeeperHiveHelper [main-EventThread]: This server instance with path null is now de-registered from ZooKeeper. 1:25:53.401 AM WARN HiveServer2 [main-EventThread]: This instance of HiveServer2 has been removed from the list of server instances available for dynamic service discovery. The last client session has ended - will shutdown now. 1:25:53.401 AM INFO CuratorFrameworkImpl [Curator-Framework-0]: backgroundOperationsLoop exiting 1:25:53.507 AM INFO ZooKeeper [main]: Session: 0x10023db368405e2 closed 1:25:53.507 AM INFO ZooKeeperHiveHelper [main]: Server instance removed from ZooKeeper. 1:25:53.507 AM INFO HiveServer2 [main]: Stopping/Disconnecting tez sessions. 1:25:53.507 AM INFO HiveServer2 [main]: Stopped tez session pool manager. 1:25:53.508 AM INFO HiveServer2 [main-EventThread]: Shutting down HiveServer2       HiveServer2 [main]: Error starting HiveServer2 on attempt 2, will retry in 60000ms org.apache.hive.service.ServiceException: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:721) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 10 more Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650968366405_0009 failed 2 times due to AM Container for appattempt_1650968366405_0009_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-27 01:25:51.885]Exception from container-launch. Container id: container_e02_1650968366405_0009_02_000001 Exit code: 1 [2022-04-27 01:25:51.890]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : [2022-04-27 01:25:51.892]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : For more detailed output, check the application tracking page: https://b7-40.lab.archivas.com:8090/cluster/app/application_1650968366405_0009 Then click on links to logs of each attempt. . Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:979) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 10 more 1:25:53.508 AM ERROR HiveServer2 [main-EventThread]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$DeRegisterWatcher.process(HiveServer2.java:617) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:77) [curator-framework-4.3.0.7.1.7.1000-141.jar:4.3.0.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:532) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] ... View more
Labels:

Unable to Start Hiveserver2 and unable to read hi...

by Explorer in Support Questions
‎04-25-2022 11:12 AM
‎04-25-2022 11:12 AM
[root]# hive SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Connecting to jdbc:hive2://b7-33.lab.archivas.com:2181,b7-40.lab.archivas.com:2181,b7-6.lab.archivas.com:2181/default;password=root;serviceDiscoveryMode=zooKeeper;ssl=true;user=root;zooKeep Error: org.apache.hive.jdbc.ZooKeeperHiveClientException: Unable to read HiveServer2 configs from ZooKeeper (state=,code=0) Beeline version 3.1.3000.7.1.7.1000-141 by Apache Hive beeline>   Logs from Hiveserver2:   SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] 2022-04-25 07:18:25: Starting HiveServer2 WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/jars/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Hive Session ID = 90fb2fa3-4cc9-41e5-9405-63342d689bc7 Hive Session ID = cc74ce55-cdd2-444b-9879-2d23d96bf2a5 22/04/25 07:18:49 ERROR server.HiveServer2: [main]: Error starting Tez sessions: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650625986491_0039 failed 2 times due to AM Container for appattempt_1650625986491_0039_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-25 07:18:48.810]Exception from container-launch. Container id: container_e01_1650625986491_0039_02_000001 Exit code: 1 [2022-04-25 07:18:48.814]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : [2022-04-25 07:18:48.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : For more detailed output, check the application tracking page: https://b7-40.lab.archivas.com:8090/cluster/app/application_1650625986491_0039 Then click on links to logs of each attempt. . Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:979) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 12 more 22/04/25 07:18:49 ERROR server.HiveServer2: [main]: Error starting Tez sessions: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650625986491_0039 failed 2 times due to AM Container for appattempt_1650625986491_0039_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-25 07:18:48.810]Exception from container-launch. Container id: container_e01_1650625986491_0039_02_000001 Exit code: 1 [2022-04-25 07:18:48.814]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : [2022-04-25 07:18:48.816]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : For more detailed output, check the application tracking page: https://b7-40.lab.archivas.com:8090/cluster/app/application_1650625986491_0039 Then click on links to logs of each attempt. . Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:979) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 12 more 22/04/25 07:18:50 ERROR server.HiveServer2: [main-EventThread]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$DeRegisterWatcher.process(HiveServer2.java:617) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:77) [curator-framework-4.3.0.7.1.7.1000-141.jar:4.3.0.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:532) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] 22/04/25 07:18:50 ERROR server.HiveServer2: [main-EventThread]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$DeRegisterWatcher.process(HiveServer2.java:617) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:77) [curator-framework-4.3.0.7.1.7.1000-141.jar:4.3.0.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:532) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] Hive Session ID = 940b7cad-fdf3-45a3-8bea-cc6a51f9fc7c Hive Session ID = 288dba15-4e1b-44d5-9088-fe7e02a63f00 22/04/25 07:20:02 ERROR server.HiveServer2: [main]: Error starting Tez sessions: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650625986491_0040 failed 2 times due to AM Container for appattempt_1650625986491_0040_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-25 07:20:02.075]Exception from container-launch. Container id: container_e01_1650625986491_0040_02_000001 Exit code: 1 [2022-04-25 07:20:02.079]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : [2022-04-25 07:20:02.080]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : For more detailed output, check the application tracking page: https://b7-40.lab.archivas.com:8090/cluster/app/application_1650625986491_0040 Then click on links to logs of each attempt. . Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:979) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 12 more 22/04/25 07:20:02 ERROR server.HiveServer2: [main]: Error starting Tez sessions: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650625986491_0040 failed 2 times due to AM Container for appattempt_1650625986491_0040_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-25 07:20:02.075]Exception from container-launch. Container id: container_e01_1650625986491_0040_02_000001 Exit code: 1 [2022-04-25 07:20:02.079]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : [2022-04-25 07:20:02.080]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : For more detailed output, check the application tracking page: https://b7-40.lab.archivas.com:8090/cluster/app/application_1650625986491_0040 Then click on links to logs of each attempt. . Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:979) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 12 more 22/04/25 07:20:03 ERROR server.HiveServer2: [main-EventThread]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$DeRegisterWatcher.process(HiveServer2.java:617) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:77) [curator-framework-4.3.0.7.1.7.1000-141.jar:4.3.0.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:532) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] 22/04/25 07:20:03 ERROR server.HiveServer2: [main-EventThread]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$DeRegisterWatcher.process(HiveServer2.java:617) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:77) [curator-framework-4.3.0.7.1.7.1000-141.jar:4.3.0.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:532) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] Hive Session ID = bbd560ad-439b-4a98-a0c0-a5909881d6df Hive Session ID = e9afcf4e-8705-435a-ba39-67724d2d445a 22/04/25 07:21:16 ERROR server.HiveServer2: [main]: Error starting Tez sessions: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650625986491_0041 failed 2 times due to AM Container for appattempt_1650625986491_0041_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-25 07:21:15.670]Exception from container-launch. Container id: container_e01_1650625986491_0041_02_000001 Exit code: 1 [2022-04-25 07:21:15.676]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : [2022-04-25 07:21:15.677]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : For more detailed output, check the application tracking page: https://b7-40.lab.archivas.com:8090/cluster/app/application_1650625986491_0041 Then click on links to logs of each attempt. . Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:979) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 12 more 22/04/25 07:21:16 ERROR server.HiveServer2: [main]: Error starting Tez sessions: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650625986491_0041 failed 2 times due to AM Container for appattempt_1650625986491_0041_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-25 07:21:15.670]Exception from container-launch. Container id: container_e01_1650625986491_0041_02_000001 Exit code: 1 [2022-04-25 07:21:15.676]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : [2022-04-25 07:21:15.677]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : For more detailed output, check the application tracking page: https://b7-40.lab.archivas.com:8090/cluster/app/application_1650625986491_0041 Then click on links to logs of each attempt. . Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:979) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 12 more 22/04/25 07:21:17 ERROR server.HiveServer2: [main-EventThread]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$DeRegisterWatcher.process(HiveServer2.java:617) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:77) [curator-framework-4.3.0.7.1.7.1000-141.jar:4.3.0.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:532) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] 22/04/25 07:21:17 ERROR server.HiveServer2: [main-EventThread]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$DeRegisterWatcher.process(HiveServer2.java:617) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:77) [curator-framework-4.3.0.7.1.7.1000-141.jar:4.3.0.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:532) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] Hive Session ID = 30474192-711d-4634-a307-0c00bb448e83 Hive Session ID = 64f38591-409a-4e59-972b-fdf11a475069 22/04/25 07:22:29 ERROR server.HiveServer2: [main]: Error starting Tez sessions: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650625986491_0042 failed 2 times due to AM Container for appattempt_1650625986491_0042_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-25 07:22:29.274]Exception from container-launch. Container id: container_e01_1650625986491_0042_02_000001 Exit code: 1 [2022-04-25 07:22:29.277]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : [2022-04-25 07:22:29.279]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : For more detailed output, check the application tracking page: https://b7-40.lab.archivas.com:8090/cluster/app/application_1650625986491_0042 Then click on links to logs of each attempt. . Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:979) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 12 more 22/04/25 07:22:29 ERROR server.HiveServer2: [main]: Error starting Tez sessions: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650625986491_0042 failed 2 times due to AM Container for appattempt_1650625986491_0042_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-25 07:22:29.274]Exception from container-launch. Container id: container_e01_1650625986491_0042_02_000001 Exit code: 1 [2022-04-25 07:22:29.277]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : [2022-04-25 07:22:29.279]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : For more detailed output, check the application tracking page: https://b7-40.lab.archivas.com:8090/cluster/app/application_1650625986491_0042 Then click on links to logs of each attempt. . Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:979) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 12 more 22/04/25 07:22:30 ERROR server.HiveServer2: [main-EventThread]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$DeRegisterWatcher.process(HiveServer2.java:617) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:77) [curator-framework-4.3.0.7.1.7.1000-141.jar:4.3.0.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:532) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] 22/04/25 07:22:30 ERROR server.HiveServer2: [main-EventThread]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$DeRegisterWatcher.process(HiveServer2.java:617) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.curator.framework.imps.NamespaceWatcher.process(NamespaceWatcher.java:77) [curator-framework-4.3.0.7.1.7.1000-141.jar:4.3.0.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:532) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507) [zookeeper-3.5.5.7.1.7.1000-141.jar:3.5.5.7.1.7.1000-141] Hive Session ID = 303f6290-e342-454f-a0fe-5cd6742c0dcd Hive Session ID = 058127d9-020d-4d06-aed6-8988dec02fd3 22/04/25 07:23:43 ERROR server.HiveServer2: [main]: Error starting Tez sessions: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650625986491_0043 failed 2 times due to AM Container for appattempt_1650625986491_0043_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-25 07:23:42.489]Exception from container-launch. Container id: container_e01_1650625986491_0043_02_000001 Exit code: 1 22/04/25 07:23:44 ERROR server.HiveServer2: [main]: Error starting HiveServer2 java.lang.Error: Max start attempts 5 exhausted at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1084) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.hive.service.ServiceException: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:721) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 9 more Caused by: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 9 more Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650625986491_0043 failed 2 times due to AM Container for appattempt_1650625986491_0043_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-25 07:23:42.489]Exception from container-launch. Container id: container_e01_1650625986491_0043_02_000001 Exit code: 1 [2022-04-25 07:23:42.493]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : [2022-04-25 07:23:42.493]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : For more detailed output, check the application tracking page: https://b7-40.lab.archivas.com:8090/cluster/app/application_1650625986491_0043 Then click on links to logs of each attempt. . Failing the application. at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:979) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.tez.client.TezClient.waitTillReady(TezClient.java:948) ~[tez-api-0.9.1.7.1.7.1000-141.jar:0.9.1.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:567) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:385) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:314) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:118) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startInitialSession(TezSessionPool.java:359) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.startUnderInitLock(TezSessionPool.java:171) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPool.start(TezSessionPool.java:123) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolManager.startPool(TezSessionPoolManager.java:115) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:822) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 9 more 22/04/25 07:23:44 ERROR server.HiveServer2: [main]: Error starting HiveServer2 java.lang.Error: Max start attempts 5 exhausted at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1084) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.access$1400(HiveServer2.java:138) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:1333) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:1177) [hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_322] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_322] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_322] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_322] at org.apache.hadoop.util.RunJar.run(RunJar.java:318) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:232) [hadoop-common-3.1.1.7.1.7.1000-141.jar:?] Caused by: org.apache.hive.service.ServiceException: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:721) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 9 more Caused by: org.apache.hive.service.ServiceException: Unable to setup tez session pool at org.apache.hive.service.server.HiveServer2.initAndStartTezSessionPoolManager(HiveServer2.java:825) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startOrReconnectTezSessions(HiveServer2.java:795) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.start(HiveServer2.java:718) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:1059) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] ... 9 more Caused by: org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1650625986491_0043 failed 2 times due to AM Container for appattempt_1650625986491_0043_000002 exited with exitCode: 1 Failing this attempt.Diagnostics: [2022-04-25 07:23:42.489]Exception from container-launch. Container id: container_e01_1650625986491_0043_02_000001 Exit code: 1 [2022-04-25 07:23:42.493]Container exited with a non-zero exit code 1. Error file: prelaunch.err. Last 4096 bytes of prelaunch.err : Last 4096 bytes of stderr : 22/04/25 07:23:44 ERROR server.HiveServer2: [shutdown-hook-0]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:439) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322] 22/04/25 07:23:44 ERROR server.HiveServer2: [shutdown-hook-0]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:439) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322] 22/04/25 07:23:44 ERROR server.HiveServer2: [shutdown-hook-0]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:439) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322] 22/04/25 07:23:44 ERROR server.HiveServer2: [shutdown-hook-0]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:439) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322] 22/04/25 07:23:44 ERROR server.HiveServer2: [shutdown-hook-0]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:439) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322] 22/04/25 07:23:44 ERROR server.HiveServer2: [shutdown-hook-0]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:439) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322] 22/04/25 07:23:44 ERROR server.HiveServer2: [shutdown-hook-0]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:439) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322] 22/04/25 07:23:44 ERROR server.HiveServer2: [shutdown-hook-0]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:439) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322] 22/04/25 07:23:44 ERROR server.HiveServer2: [shutdown-hook-0]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:439) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322] 22/04/25 07:23:44 ERROR server.HiveServer2: [shutdown-hook-0]: Error stopping schq java.lang.IllegalStateException: The current ScheduledQueryExecutionService INSTANCE is invalid at org.apache.hadoop.hive.ql.scheduled.ScheduledQueryExecutionService.close(ScheduledQueryExecutionService.java:312) ~[hive-exec-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.stop(HiveServer2.java:892) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at org.apache.hive.service.server.HiveServer2.lambda$init$0(HiveServer2.java:439) ~[hive-service-3.1.3000.7.1.7.1000-141.jar:3.1.3000.7.1.7.1000-141] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_322] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_322] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_322] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_322]   ... View more
Labels:
Contact Me
Online
Offline
Last Visited
‎07-05-2022 07:43 AM
Community Statistics
Member Since ‎04-25-2022 11:01 AM
Last Visited ‎07-05-2022 07:43 AM
Posts 4