This time we did show table stats and show column stats on the table before the issue ( just after the restart and before running invalidate metadata) and then after the restart and we do not notice any difference in output but problem reoccurred this time also.  To understand the sequence of events : (1) Admin team does patching and restart the cluster (2) application team run a distinct year query and sees out of 31 only 27 rows are present and 4 of the partitions are missing.  (3) as a support team, I run show stats which shows all 31 partitions and number of rows in partitions is correct and size also. (4) we run invalidate metadata and refresh (5) support team takes show stats which has same result (6) app team runs they query again and they able to see all 31 partitions.   we will work with vendor to get further help.  ... View more

@rachida_el-hamm, did @Shelton's response help you resolve your issue?  If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.   ... View more

@Anup123  I responded to a similar question see SSL Sqoop If already have an SSL cert file, then you can generate your own JKS file and import your cert into your jks. ... View more

@Shelton - This is how the logs starts. It would be great if you can help.   2021-09-01T19:54:27,931 WARN [main] session.SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory. 2021-09-01T19:54:28,978 INFO [main] hive.metastore: Trying to connect to metastore with URI thrift://192.168.15.77:1465 2021-09-01T19:54:29,100 INFO [main] hive.metastore: Opened a connection to metastore, current connections: 1 2021-09-01T19:54:29,101 INFO [main] hive.metastore: Connected to metastore. 2021-09-01T19:54:29,236 WARN [main] metastore.RetryingMetaStoreClient: MetaStoreClient lost connection. Attempting to reconnect (1 of 1) after 1s. getAllFunctions org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_all_functions(ThriftHiveMetastore.java:3845) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_all_functions(ThriftHiveMetastore.java:3833) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getAllFunctions(HiveMetaStoreClient.java:2399) ~[hive-exec-2.3.8.jar:2.3.8] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_172] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_172] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_172] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172] at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:173) ~[hive-exec-2.3.8.jar:2.3.8] at com.sun.proxy.$Proxy34.getAllFunctions(Unknown Source) ~[?:?] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_172] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_172] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_172] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2336) ~[hive-exec-2.3.8.jar:2.3.8] at com.sun.proxy.$Proxy34.getAllFunctions(Unknown Source) ~[?:?] at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3894) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:388) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:332) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:312) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:288) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:917) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:881) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1687) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:130) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.cli.CLIService.init(CLIService.java:114) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.CompositeService.init(CompositeService.java:59) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:142) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:607) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:100) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:855) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:724) ~[hive-service-2.3.8.jar:2.3.8] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_172] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_172] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_172] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172] at org.apache.hadoop.util.RunJar.run(RunJar.java:221) ~[hadoop-common-2.7.3.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:136) ~[hadoop-common-2.7.3.jar:?] 2021-09-01T19:54:30,247 INFO [main] hive.metastore: Closed a connection to metastore, current connections: 0 2021-09-01T19:54:30,248 INFO [main] hive.metastore: Trying to connect to metastore with URI thrift://192.168.15.77:1465 2021-09-01T19:54:30,249 INFO [main] hive.metastore: Opened a connection to metastore, current connections: 1 2021-09-01T19:54:30,250 INFO [main] hive.metastore: Connected to metastore. 2021-09-01T19:54:30,284 INFO [main] server.HiveServer2: Shutting down HiveServer2 2021-09-01T19:54:30,283 WARN [main] metadata.Hive: Failed to register all functions. org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.thrift.transport.TTransportException at org.apache.hadoop.hive.ql.metadata.Hive.getAllFunctions(Hive.java:3897) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.metadata.Hive.reloadFunctions(Hive.java:248) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.metadata.Hive.registerAllFunctionsOnce(Hive.java:231) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.metadata.Hive.<init>(Hive.java:388) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.metadata.Hive.create(Hive.java:332) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.metadata.Hive.getInternal(Hive.java:312) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.metadata.Hive.get(Hive.java:288) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:917) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:881) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:1687) ~[hive-exec-2.3.8.jar:2.3.8] at org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:130) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.cli.CLIService.init(CLIService.java:114) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.CompositeService.init(CompositeService.java:59) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:142) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:607) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:100) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:855) ~[hive-service-2.3.8.jar:2.3.8] at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:724) ~[hive-service-2.3.8.jar:2.3.8] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_172] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_172] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_172] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_172] at org.apache.hadoop.util.RunJar.run(RunJar.java:221) ~[hadoop-common-2.7.3.jar:?] at org.apache.hadoop.util.RunJar.main(RunJar.java:136) ~[hadoop-common-2.7.3.jar:?] Caused by: org.apache.thrift.transport.TTransportException ... View more

Thanks you very much @Shelton. You are the best! ... View more

@Shelton   I have already executed steps that you have provided in the link but still ambari services rest api url is not kerberized .   is there any other configuration required ? or ambari services rest api url do not support kerberos authentication ? ... View more

@mike_bronson7  Can you share your capacity scheduler , total memory and vcores configs ?    ... View more

@Shelton  thanks for the reply. can able to set every Wednesday to run, But not able to schedule every other Wednesday. ... View more

@Vinay1991  I mentioned the logs below.You will need definitely the ZK ZKFailoverController logs and NameNode logs ... View more

@Buithuy96  First and foremost re-running these steps won't do damage to your cluster I assure you. What you have is purely a permissions issue java.sql.SQLException: Access denied for user 'ambari'@'mtnode.hdp.vn' (using password: YES) Revalidate the MySQL connector # yum install -y mysql-connector-java # ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar Re-run the Ambari user setup CREATE USER 'ambari'@'%' IDENTIFIED BY 'aCtct@123'; CREATE USER 'ambari'@'localhost' IDENTIFIED BY 'Ctct@123'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'%'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'localhost'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'mtnode.hdp.vn'; GRANT ALL PRIVILEGES ON *.* TO 'ambari'@'mtnode.hdp.vn' IDENTIFIED BY 'Ctct@123'; FLUSH PRIVILEGES; Try restarting Ambari while tailing the ambari-server.log and share the contents, first reset the log before starting Ambari, to ensure you have a minimum log to delve through # truncate --size 0 /var/logs/ambari-server/ambari-server.log Restart your Ambari server and tail the logs # tail -f /var/logs/ambari-server/ambari-server.log Share the status ... View more