First of all , It is not recommended to use the same location for both internal and external tables. Internal tables in Hive are native tables that are fully controlled by Hive itself. External tables, on the other hand, can be accessed by other components such as Spark, Impala, and File system operations,.....etc. Since External tables are used by other components, their corresponding locations need to be relied upon. To read the files and obtain the count, Hive launches a MapReduce job for external tables. It is recommended to use Managed tables if other components are not utilizing the corresponding table.  ... View more

From the attached console output noticed AM failed to submit the DAG. ERROR : Failed to execute tez graph. org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1730584072947_0051 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1730584072947_0051_000001 exited with exitCode: 1  Please check the appLogs application_1730584072947_0051 to identify the root cause of the failure.  ... View more

The job failed with an OutOfMemoryError (OOME) at the child task attempt level, as indicated by the stacktrace. It was observed that certain mapreduce properties have been set, which may potentially overwrite the hive.tez.container.size property.   SET mapreduce.map.java.opts=-Xmx3686m; SET mapreduce.reduce.java.opts=-Xmx3686m; SET mapred.child.java.opts=-Xmx10g;​   It is recommended to validate the yarn appLogs to confirm if the child task attempts were launched with 80% of the hive.tez.container.size. If not, it is advised to remove the mapreduce configurations and try re-running the job. Before re-running the query, it is suggested to collect statistics for all the source tables. This will assist the optimizer in creating a better execution plan. ... View more

If you suspect that TEZ-4032 is the cause, consider upgrading your cluster to CDP 7 and testing it again, as it has been backported in CDP 7. ... View more

Ideally below should work.  use default;show tables; Please check HiveServer2, HMS logs and share the stack-trace.  To identify the RootCause.  ... View more

The query seems to be failing during the compilation phase, indicating a possible issue with its syntax. Error: Error while compiling statement: FAILED: ParseException line 1:11 missing EOF at ';' near 'default' (state=42000,code=40000) It is important to validate the SQL syntax of the query to identify any potential syntax errors that may be causing the problem. Another point to consider is ensuring that the column names in the query match with those in the source and target tables, as any mismatch can lead to errors. ... View more

The below error usually occurs when there is a mismatch between the versions of Tez and Hive. Validate the compatibility.  Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 112 Serialization trace: conf (org.apache.hadoop.hive.ql.exec.TableScanOperator) aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:159) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:758) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:188) at org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:117) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:129) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:877) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:183) at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:235) at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:42) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:796) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:221) at org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:124) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:129) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:774) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213) at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectByKryo(SerializationUtilities.java:838) at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:745) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:489) ... 19 more Additionally, it is important to validate if any auxiliary jars are being used and ensure that these jars do not conflict with the jars in the Hive classpath. if two versions of same jars are present in the classapth , which also creates a problem in the serialization. ... View more

Cloudera Manager offers configurations and tools to optimize the usage of Tez as an execution engine for big data workloads, particularly when using Hive. The Tez CM-UI in Cloudera Manager allows for fine-tuning of Tez configurations to meet specific workload requirements, including adjusting Tez container sizes, memory settings, and parallelism levels. These Tez configurations can be applied cluster-wide, specifically for Hive. Please verify that the HIVE and Hive_ON_TEZ services are active and functioning properly. Similarly, the Core configuration applies to the entire cluster and provides an interface to change configurations cluster-wide. Ensure that all services are running smoothly. If any particular service appears as red, investigate the individual service logs for further troubleshooting. ... View more

As mentioned in the previous comment , it's advisable to use the static value.  Example -  CREATE TABLE test_table ( id INT, name STRING ) PARTITIONED BY (START_DATE STRING); -- Add sample partitions for testing ALTER TABLE test_table ADD PARTITION (START_DATE = '2023-09-01'); ALTER TABLE test_table ADD PARTITION (START_DATE = '2023-08-31'); -- This should be the target partition to drop ALTER TABLE test_table ADD PARTITION (START_DATE = '2023-08-30'); compute the date in the shell and pass it to Hive. # Set the base date and calculate the target date (13 days before the base date) base_date="2023-09-13" partition_date=$(date -d "$base_date - 13 days" +%Y-%m-%d) # Log the computed date (optional) echo "Dropping partition for date: $partition_date" # Execute the Hive command to drop the partition beeline -n hive -p hive -e "ALTER TABLE test_table DROP PARTITION (START_DATE = '$partition_date');show partitions test_table;" Sample Run  [root@ccycloud-2.blesc-15238.root.comops.site ~]# # Set the base date and calculate the target date (13 days before the base date) [root@ccycloud-2.blesc-15238.root.comops.site ~]# base_date="2023-09-13" [root@ccycloud-2.blesc-15238.root.comops.site ~]# partition_date=$(date -d "$base_date - 13 days" +%Y-%m-%d) [root@ccycloud-2.blesc-15238.root.comops.site ~]# [root@ccycloud-2.blesc-15238.root.comops.site ~]# # Log the computed date (optional) [root@ccycloud-2.blesc-15238.root.comops.site ~]# echo "Dropping partition for date: $partition_date" Dropping partition for date: 2023-08-31 [root@ccycloud-2.blesc-15238.root.comops.site ~]# [root@ccycloud-2.blesc-15238.root.comops.site ~]# # Execute the Hive command to drop the partition [root@ccycloud-2.blesc-15238.root.comops.site ~]# beeline -n hive -p hive -e "ALTER TABLE test_table DROP PARTITION (START_DATE = '$partition_date');show partitions test_table;" SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p9.52289703/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p9.52289703/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p9.52289703/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p9.52289703/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Connecting to jdbc:hive2://ccycloud-2.blesc-15238.root.comops.site:2181/default;password=hive;principal=hive/_HOST@ROOT.COMOPS.SITE;serviceDiscoveryMode=zooKeeper;user=hive;zooKeeperNamespace=hiveserver2 24/11/13 15:08:10 [main]: INFO jdbc.HiveConnection: Connected to ccycloud-2.blesc-15238.root.comops.site:10000 Connected to: Apache Hive (version 3.1.3000.7.1.9.9-1) Driver: Hive JDBC (version 3.1.3000.7.1.9.9-1) Transaction isolation: TRANSACTION_REPEATABLE_READ INFO : Compiling command(queryId=hive_20241113150811_5d177453-0f69-4fee-baa3-9d2a6e82c828): ALTER TABLE test_table DROP PARTITION (START_DATE = '2023-08-31') INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20241113150811_5d177453-0f69-4fee-baa3-9d2a6e82c828); Time taken: 1.076 seconds INFO : Executing command(queryId=hive_20241113150811_5d177453-0f69-4fee-baa3-9d2a6e82c828): ALTER TABLE test_table DROP PARTITION (START_DATE = '2023-08-31') INFO : Starting task [Stage-0:DDL] in serial mode INFO : Dropped the partition start_date=2023-08-31 INFO : Completed executing command(queryId=hive_20241113150811_5d177453-0f69-4fee-baa3-9d2a6e82c828); Time taken: 0.251 seconds INFO : OK No rows affected (1.417 seconds) INFO : Compiling command(queryId=hive_20241113150812_da9db0ea-d18d-4016-9a3f-3da3535caf13): show partitions test_table INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:partition, type:string, comment:from deserializer)], properties:null) INFO : Completed compiling command(queryId=hive_20241113150812_da9db0ea-d18d-4016-9a3f-3da3535caf13); Time taken: 0.039 seconds INFO : Executing command(queryId=hive_20241113150812_da9db0ea-d18d-4016-9a3f-3da3535caf13): show partitions test_table INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20241113150812_da9db0ea-d18d-4016-9a3f-3da3535caf13); Time taken: 0.053 seconds INFO : OK +------------------------+ | partition | +------------------------+ | start_date=2023-08-30 | | start_date=2023-09-01 | +------------------------+ 2 rows selected (0.185 seconds) Beeline version 3.1.3000.7.1.9.9-1 by Apache Hive Closing: 0: jdbc:hive2://ccycloud-2.blesc-15238.root.comops.site:2181/default;password=hive;principal=hive/_HOST@ROOT.COMOPS.SITE;serviceDiscoveryMode=zooKeeper;user=hive;zooKeeperNamespace=hiveserver2 [root@ccycloud-2.blesc-15238.root.comops.site ~]# ... View more