In Hive ACID tables, the base/delta files are associated with a writeID. My assumption is the write ID has changed or got removed in the production HMS, so hive does not recognize the data file. So, we will have to follow the workaround that I mentioned in my last update.  ... View more

@AhXian the metadata is still not updated, as I see the following: ,numFiles ,0 ,numRows ,0 ,rawDataSize ,0 ,totalSize ,0 Let's try the following commands one after the other: MSCK REPAIR raw_zone_sit.jxtemp SYNC PARTITIONS; ANALYZE TABLE raw_zone_sit.jxtemp PARTITION (dl_created_yr='2023', dl_created_mth='12') COMPUTE STATISTICS; ANALYZE TABLE raw_zone_sit.jxtemp PARTITION (dl_created_yr='2023', dl_created_mth='12') COMPUTE STATISTICS for COLUMNS; See if the describe command still reflects 0 files. Also, could you collect the output of the following command from the backend metastore database? select * from NEXT_WRITE_ID where NWI_TABLE='jxtemp'; ... View more

@AhXian These ORC tables we are talking about, are they ACID? If we copy data files to ACID table location, and run ANALYZE...COMPUTE STATISTICS, that alone might not resolve the issue. We would like for you to share an example. Please share the following command output: describe formatted table_name partition(dl_created_yr='2023', dl_created_mth='12') &HDFS command o/p hdfs dfs -ls -R <partition location from the above command> ... View more

@ldylag You may find the DDLs under /opt/cloudera/parcels/CDH/lib/hive/scripts/metastore/upgrade/<db type> ... View more

@zhuodongLi It will be difficult to find the cause of the issue from the error message in the screenshot. It will be best if you could create a support case with the YARN app log of application_1719542914905_0002, and HS2 logs covering the job run period. ... View more

@kaif This is non-Cloudera Hive you are testing. But let me share my two cents.  You are using 'org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdConfOnlyAuthorizerFactory' as the auth manager. You may want to use the following auth provider that enables you to use GRANT/REVOKE statements. hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactory ...and use: GRANT ROLE admin TO USER kaif; SET ROLE admin; ... View more

@kaif Hive does have information_schema database, that you can access from beeline or Hue. You also have SYS db that fetches different information from the RDBMS that is used to store hive metadata.  Here's the DDLs of those tables, and this will also give you some idea on what data they store. ... View more

@ldylag No, it is not expected behavior. We expect the tables to be created in the 'public' schema under 'metastore' database. ... View more

@hadoopranger It seems like a transaction is in failed state. a drop database call probably did not complete successfully. Did we make config changes in postgres while Hive service was running? That could lead to data corruption.  I notice the following error: Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block org.datanucleus.exceptions.NucleusDataStoreException: Cannot get Object result for param = 6 for column ""MV_CREATION_METADATA"."TXN_LIST"" : ERROR: current transaction is aborted, commands ignored until end of transaction block You may get the out of the following command and see which transaction is in idle but aborted state.   select * from pg_stat_activity; Then try to terminate the transaction or perform a ROLLBACK. Take a backup of the database before making these changes to be on the safer side. ... View more