About ggangadharan

ggangadharan · ‎12-26-2024

The error message Invalid SessionHandle: SessionHandle commonly occurs in Hive when there is an issue with the session handle being used. A session handle in Hive is a unique identifier for a session created when a user connects to Hive, used to maintain the state and context of the session. One possible scenario for this error is when a table contains a large number of records and the cluster has multiple HS2 instances. If Knox is used to connect to Hive, Knox might connect to one HS2 and run a query. However, due to the large number of records, the query takes longer to process. If the connection times out from Knox's end and reconnects to another HS2, the query might fail with the "Invalid SessionHandle" error. To investigate this scenario, it is recommended to check the HS2 logs and Knox logs. Additionally, to determine why the query is running long, checking the HS2 and appLogs for any yarn job initiated by HS2 can provide further insights.

ggangadharan · ‎11-28-2024

First of all , It is not recommended to use the same location for both internal and external tables. Internal tables in Hive are native tables that are fully controlled by Hive itself. External tables, on the other hand, can be accessed by other components such as Spark, Impala, and File system operations,.....etc. Since External tables are used by other components, their corresponding locations need to be relied upon. To read the files and obtain the count, Hive launches a MapReduce job for external tables. It is recommended to use Managed tables if other components are not utilizing the corresponding table.

ggangadharan · ‎11-25-2024

From the attached console output noticed AM failed to submit the DAG. ERROR : Failed to execute tez graph. org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1730584072947_0051 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1730584072947_0051_000001 exited with exitCode: 1 Please check the appLogs application_1730584072947_0051 to identify the root cause of the failure.

ggangadharan · ‎11-22-2024

The job failed with an OutOfMemoryError (OOME) at the child task attempt level, as indicated by the stacktrace. It was observed that certain mapreduce properties have been set, which may potentially overwrite the hive.tez.container.size property. SET mapreduce.map.java.opts=-Xmx3686m; SET mapreduce.reduce.java.opts=-Xmx3686m; SET mapred.child.java.opts=-Xmx10g; It is recommended to validate the yarn appLogs to confirm if the child task attempts were launched with 80% of the hive.tez.container.size. If not, it is advised to remove the mapreduce configurations and try re-running the job. Before re-running the query, it is suggested to collect statistics for all the source tables. This will assist the optimizer in creating a better execution plan.

ggangadharan · ‎11-22-2024

If you suspect that TEZ-4032 is the cause, consider upgrading your cluster to CDP 7 and testing it again, as it has been backported in CDP 7.

ggangadharan · ‎11-21-2024

Ideally below should work. use default;show tables; Please check HiveServer2, HMS logs and share the stack-trace. To identify the RootCause.

ggangadharan · ‎11-18-2024

The query seems to be failing during the compilation phase, indicating a possible issue with its syntax. Error: Error while compiling statement: FAILED: ParseException line 1:11 missing EOF at ';' near 'default' (state=42000,code=40000) It is important to validate the SQL syntax of the query to identify any potential syntax errors that may be causing the problem. Another point to consider is ensuring that the column names in the query match with those in the source and target tables, as any mismatch can lead to errors.

ggangadharan · ‎11-15-2024

The below error usually occurs when there is a mismatch between the versions of Tez and Hive. Validate the compatibility. Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 112 Serialization trace: conf (org.apache.hadoop.hive.ql.exec.TableScanOperator) aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:159) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:758) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:188) at org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:117) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:129) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:877) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:183) at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:235) at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:42) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:796) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:221) at org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:124) at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:129) at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:774) at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213) at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectByKryo(SerializationUtilities.java:838) at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:745) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:489) ... 19 more Additionally, it is important to validate if any auxiliary jars are being used and ensure that these jars do not conflict with the jars in the Hive classpath. if two versions of same jars are present in the classapth , which also creates a problem in the serialization.

ggangadharan · ‎11-13-2024

Cloudera Manager offers configurations and tools to optimize the usage of Tez as an execution engine for big data workloads, particularly when using Hive. The Tez CM-UI in Cloudera Manager allows for fine-tuning of Tez configurations to meet specific workload requirements, including adjusting Tez container sizes, memory settings, and parallelism levels. These Tez configurations can be applied cluster-wide, specifically for Hive. Please verify that the HIVE and Hive_ON_TEZ services are active and functioning properly. Similarly, the Core configuration applies to the entire cluster and provides an interface to change configurations cluster-wide. Ensure that all services are running smoothly. If any particular service appears as red, investigate the individual service logs for further troubleshooting.

ggangadharan · ‎11-13-2024

As mentioned in the previous comment , it's advisable to use the static value. Example - CREATE TABLE test_table ( id INT, name STRING ) PARTITIONED BY (START_DATE STRING); -- Add sample partitions for testing ALTER TABLE test_table ADD PARTITION (START_DATE = '2023-09-01'); ALTER TABLE test_table ADD PARTITION (START_DATE = '2023-08-31'); -- This should be the target partition to drop ALTER TABLE test_table ADD PARTITION (START_DATE = '2023-08-30'); compute the date in the shell and pass it to Hive. # Set the base date and calculate the target date (13 days before the base date) base_date="2023-09-13" partition_date=$(date -d "$base_date - 13 days" +%Y-%m-%d) # Log the computed date (optional) echo "Dropping partition for date: $partition_date" # Execute the Hive command to drop the partition beeline -n hive -p hive -e "ALTER TABLE test_table DROP PARTITION (START_DATE = '$partition_date');show partitions test_table;" Sample Run [root@ccycloud-2.blesc-15238.root.comops.site ~]# # Set the base date and calculate the target date (13 days before the base date) [root@ccycloud-2.blesc-15238.root.comops.site ~]# base_date="2023-09-13" [root@ccycloud-2.blesc-15238.root.comops.site ~]# partition_date=$(date -d "$base_date - 13 days" +%Y-%m-%d) [root@ccycloud-2.blesc-15238.root.comops.site ~]# [root@ccycloud-2.blesc-15238.root.comops.site ~]# # Log the computed date (optional) [root@ccycloud-2.blesc-15238.root.comops.site ~]# echo "Dropping partition for date: $partition_date" Dropping partition for date: 2023-08-31 [root@ccycloud-2.blesc-15238.root.comops.site ~]# [root@ccycloud-2.blesc-15238.root.comops.site ~]# # Execute the Hive command to drop the partition [root@ccycloud-2.blesc-15238.root.comops.site ~]# beeline -n hive -p hive -e "ALTER TABLE test_table DROP PARTITION (START_DATE = '$partition_date');show partitions test_table;" SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p9.52289703/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p9.52289703/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] WARNING: Use "yarn jar" to launch YARN applications. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p9.52289703/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p9.52289703/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory] Connecting to jdbc:hive2://ccycloud-2.blesc-15238.root.comops.site:2181/default;password=hive;principal=hive/_HOST@ROOT.COMOPS.SITE;serviceDiscoveryMode=zooKeeper;user=hive;zooKeeperNamespace=hiveserver2 24/11/13 15:08:10 [main]: INFO jdbc.HiveConnection: Connected to ccycloud-2.blesc-15238.root.comops.site:10000 Connected to: Apache Hive (version 3.1.3000.7.1.9.9-1) Driver: Hive JDBC (version 3.1.3000.7.1.9.9-1) Transaction isolation: TRANSACTION_REPEATABLE_READ INFO : Compiling command(queryId=hive_20241113150811_5d177453-0f69-4fee-baa3-9d2a6e82c828): ALTER TABLE test_table DROP PARTITION (START_DATE = '2023-08-31') INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null) INFO : Completed compiling command(queryId=hive_20241113150811_5d177453-0f69-4fee-baa3-9d2a6e82c828); Time taken: 1.076 seconds INFO : Executing command(queryId=hive_20241113150811_5d177453-0f69-4fee-baa3-9d2a6e82c828): ALTER TABLE test_table DROP PARTITION (START_DATE = '2023-08-31') INFO : Starting task [Stage-0:DDL] in serial mode INFO : Dropped the partition start_date=2023-08-31 INFO : Completed executing command(queryId=hive_20241113150811_5d177453-0f69-4fee-baa3-9d2a6e82c828); Time taken: 0.251 seconds INFO : OK No rows affected (1.417 seconds) INFO : Compiling command(queryId=hive_20241113150812_da9db0ea-d18d-4016-9a3f-3da3535caf13): show partitions test_table INFO : Semantic Analysis Completed (retrial = false) INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:partition, type:string, comment:from deserializer)], properties:null) INFO : Completed compiling command(queryId=hive_20241113150812_da9db0ea-d18d-4016-9a3f-3da3535caf13); Time taken: 0.039 seconds INFO : Executing command(queryId=hive_20241113150812_da9db0ea-d18d-4016-9a3f-3da3535caf13): show partitions test_table INFO : Starting task [Stage-0:DDL] in serial mode INFO : Completed executing command(queryId=hive_20241113150812_da9db0ea-d18d-4016-9a3f-3da3535caf13); Time taken: 0.053 seconds INFO : OK +------------------------+ | partition | +------------------------+ | start_date=2023-08-30 | | start_date=2023-09-01 | +------------------------+ 2 rows selected (0.185 seconds) Beeline version 3.1.3000.7.1.9.9-1 by Apache Hive Closing: 0: jdbc:hive2://ccycloud-2.blesc-15238.root.comops.site:2181/default;password=hive;principal=hive/_HOST@ROOT.COMOPS.SITE;serviceDiscoveryMode=zooKeeper;user=hive;zooKeeperNamespace=hiveserver2 [root@ccycloud-2.blesc-15238.root.comops.site ~]#

Online	Offline
Last Visited	‎01-02-2025 09:06 AM

Member Since	‎09-16-2021 02:45 AM
Last Visited	‎01-02-2025 09:06 AM
Posts	336
Kudos received	53

Cloudera Community

Re: Hive Job - OutOfMemoryError: Java heap space

Re: Insert into table test values('a', 'b'); not w...

Re: how to drop partition table using date_add fun...

Re: Issue with Hive HQL insert query - KryoExcepti...

Re: Error when do an alter table change column on ...

Re: keep alive - Error message from Server: Inva...

Re: Why 'count()' query of hive external table ...

Re: Error while trying to perform insert operation...

Re: Hive Job - OutOfMemoryError: Java heap space

Re: Hive can not insert after enable HDFS Federati...

Re: Hive beeline multiple queries not working in c...

Re: Hive beeline multiple queries not working in c...

Re: Insert into table test values('a', 'b'); not w...

Re: Core configuration service and tez is not wor...

Re: how to drop partition table using date_add fun...