Member since
09-16-2021
394
Posts
53
Kudos Received
33
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
136 | 04-23-2025 12:46 AM | |
165 | 03-28-2025 03:47 AM | |
820 | 02-10-2025 08:58 AM | |
406 | 02-04-2025 06:33 AM | |
499 | 02-04-2025 05:55 AM |
11-21-2024
10:14 PM
Ideally below should work. use default;show tables; Please check HiveServer2, HMS logs and share the stack-trace. To identify the RootCause.
... View more
11-18-2024
05:08 AM
1 Kudo
The query seems to be failing during the compilation phase, indicating a possible issue with its syntax. Error: Error while compiling statement: FAILED: ParseException line 1:11 missing EOF at ';' near 'default' (state=42000,code=40000) It is important to validate the SQL syntax of the query to identify any potential syntax errors that may be causing the problem. Another point to consider is ensuring that the column names in the query match with those in the source and target tables, as any mismatch can lead to errors.
... View more
11-15-2024
06:38 AM
1 Kudo
The below error usually occurs when there is a mismatch between the versions of Tez and Hive. Validate the compatibility. Caused by: org.apache.hive.com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 112
Serialization trace:
conf (org.apache.hadoop.hive.ql.exec.TableScanOperator)
aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
at org.apache.hive.com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:159)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:758)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClass(SerializationUtilities.java:188)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:117)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:129)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:877)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readClassAndObject(SerializationUtilities.java:183)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:235)
at org.apache.hive.com.esotericsoftware.kryo.serializers.MapSerializer.read(MapSerializer.java:42)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:796)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:221)
at org.apache.hive.com.esotericsoftware.kryo.serializers.ReflectField.read(ReflectField.java:124)
at org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:129)
at org.apache.hive.com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:774)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities$KryoWithHooks.readObject(SerializationUtilities.java:213)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializeObjectByKryo(SerializationUtilities.java:838)
at org.apache.hadoop.hive.ql.exec.SerializationUtilities.deserializePlan(SerializationUtilities.java:745)
at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:489)
... 19 more Additionally, it is important to validate if any auxiliary jars are being used and ensure that these jars do not conflict with the jars in the Hive classpath. if two versions of same jars are present in the classapth , which also creates a problem in the serialization.
... View more
11-13-2024
11:09 PM
1 Kudo
Cloudera Manager offers configurations and tools to optimize the usage of Tez as an execution engine for big data workloads, particularly when using Hive. The Tez CM-UI in Cloudera Manager allows for fine-tuning of Tez configurations to meet specific workload requirements, including adjusting Tez container sizes, memory settings, and parallelism levels. These Tez configurations can be applied cluster-wide, specifically for Hive. Please verify that the HIVE and Hive_ON_TEZ services are active and functioning properly. Similarly, the Core configuration applies to the entire cluster and provides an interface to change configurations cluster-wide. Ensure that all services are running smoothly. If any particular service appears as red, investigate the individual service logs for further troubleshooting.
... View more
11-13-2024
07:12 AM
As mentioned in the previous comment , it's advisable to use the static value. Example - CREATE TABLE test_table (
id INT,
name STRING
)
PARTITIONED BY (START_DATE STRING);
-- Add sample partitions for testing
ALTER TABLE test_table ADD PARTITION (START_DATE = '2023-09-01');
ALTER TABLE test_table ADD PARTITION (START_DATE = '2023-08-31'); -- This should be the target partition to drop
ALTER TABLE test_table ADD PARTITION (START_DATE = '2023-08-30'); compute the date in the shell and pass it to Hive. # Set the base date and calculate the target date (13 days before the base date)
base_date="2023-09-13"
partition_date=$(date -d "$base_date - 13 days" +%Y-%m-%d)
# Log the computed date (optional)
echo "Dropping partition for date: $partition_date"
# Execute the Hive command to drop the partition
beeline -n hive -p hive -e "ALTER TABLE test_table DROP PARTITION (START_DATE = '$partition_date');show partitions test_table;" Sample Run [root@ccycloud-2.blesc-15238.root.comops.site ~]# # Set the base date and calculate the target date (13 days before the base date)
[root@ccycloud-2.blesc-15238.root.comops.site ~]# base_date="2023-09-13"
[root@ccycloud-2.blesc-15238.root.comops.site ~]# partition_date=$(date -d "$base_date - 13 days" +%Y-%m-%d)
[root@ccycloud-2.blesc-15238.root.comops.site ~]#
[root@ccycloud-2.blesc-15238.root.comops.site ~]# # Log the computed date (optional)
[root@ccycloud-2.blesc-15238.root.comops.site ~]# echo "Dropping partition for date: $partition_date"
Dropping partition for date: 2023-08-31
[root@ccycloud-2.blesc-15238.root.comops.site ~]#
[root@ccycloud-2.blesc-15238.root.comops.site ~]# # Execute the Hive command to drop the partition
[root@ccycloud-2.blesc-15238.root.comops.site ~]# beeline -n hive -p hive -e "ALTER TABLE test_table DROP PARTITION (START_DATE = '$partition_date');show partitions test_table;"
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p9.52289703/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p9.52289703/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
WARNING: Use "yarn jar" to launch YARN applications.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p9.52289703/jars/log4j-slf4j-impl-2.18.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-7.1.9-1.cdh7.1.9.p9.52289703/jars/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Connecting to jdbc:hive2://ccycloud-2.blesc-15238.root.comops.site:2181/default;password=hive;principal=hive/_HOST@ROOT.COMOPS.SITE;serviceDiscoveryMode=zooKeeper;user=hive;zooKeeperNamespace=hiveserver2
24/11/13 15:08:10 [main]: INFO jdbc.HiveConnection: Connected to ccycloud-2.blesc-15238.root.comops.site:10000
Connected to: Apache Hive (version 3.1.3000.7.1.9.9-1)
Driver: Hive JDBC (version 3.1.3000.7.1.9.9-1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
INFO : Compiling command(queryId=hive_20241113150811_5d177453-0f69-4fee-baa3-9d2a6e82c828): ALTER TABLE test_table DROP PARTITION (START_DATE = '2023-08-31')
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:null, properties:null)
INFO : Completed compiling command(queryId=hive_20241113150811_5d177453-0f69-4fee-baa3-9d2a6e82c828); Time taken: 1.076 seconds
INFO : Executing command(queryId=hive_20241113150811_5d177453-0f69-4fee-baa3-9d2a6e82c828): ALTER TABLE test_table DROP PARTITION (START_DATE = '2023-08-31')
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Dropped the partition start_date=2023-08-31
INFO : Completed executing command(queryId=hive_20241113150811_5d177453-0f69-4fee-baa3-9d2a6e82c828); Time taken: 0.251 seconds
INFO : OK
No rows affected (1.417 seconds)
INFO : Compiling command(queryId=hive_20241113150812_da9db0ea-d18d-4016-9a3f-3da3535caf13): show partitions test_table
INFO : Semantic Analysis Completed (retrial = false)
INFO : Created Hive schema: Schema(fieldSchemas:[FieldSchema(name:partition, type:string, comment:from deserializer)], properties:null)
INFO : Completed compiling command(queryId=hive_20241113150812_da9db0ea-d18d-4016-9a3f-3da3535caf13); Time taken: 0.039 seconds
INFO : Executing command(queryId=hive_20241113150812_da9db0ea-d18d-4016-9a3f-3da3535caf13): show partitions test_table
INFO : Starting task [Stage-0:DDL] in serial mode
INFO : Completed executing command(queryId=hive_20241113150812_da9db0ea-d18d-4016-9a3f-3da3535caf13); Time taken: 0.053 seconds
INFO : OK
+------------------------+
| partition |
+------------------------+
| start_date=2023-08-30 |
| start_date=2023-09-01 |
+------------------------+
2 rows selected (0.185 seconds)
Beeline version 3.1.3000.7.1.9.9-1 by Apache Hive
Closing: 0: jdbc:hive2://ccycloud-2.blesc-15238.root.comops.site:2181/default;password=hive;principal=hive/_HOST@ROOT.COMOPS.SITE;serviceDiscoveryMode=zooKeeper;user=hive;zooKeeperNamespace=hiveserver2
[root@ccycloud-2.blesc-15238.root.comops.site ~]#
... View more
11-10-2024
11:19 PM
1 Kudo
Generally, a NoSuchMethodError occurs when there is a mismatch in the classpath. To resolve this issue, please verify the classpaths of both HS2 and Hadoop, to ensure that there are no incorrect versions of Hadoop libraries present. If you are not able to identify any mismatched jar, please add the -verboseJVM argument to both HS2 and hive.tez.java.opts. This will help validate the classes loaded as part of the query and provide additional information.
... View more
11-10-2024
11:05 PM
1 Kudo
Please review the HiveServer2 logs for the specific timeframe and provide us with the complete Stacktrace of the failed query, including the cluster version and query details, as well as the DDL statements for the source and target tables.
... View more
11-06-2024
02:58 AM
1 Kudo
It's not recommended to disable kerberos . At the same time it's not direct way to disable kerberos. You can try to follow the below community post to check if that helps. https://community.cloudera.com/t5/Support-Questions/Disabling-Kerberos/m-p/19934#M38077 https://community.cloudera.com/t5/Customer/How-to-Revert-Kerberos-Configuration-from-a-Cloudera-Managed/ta-p/73464
... View more
10-29-2024
12:42 AM
1 Kudo
To access Kafka topics in a security-enabled cluster, please follow the steps outlined below. kinit with appropriate user. kinit -kt <user> <principal> Create jaas.conf file. KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
}; Export the jaas.conf file in KAFKA_OPTS environment variable. export KAFKA_OPTS="-Djava.security.auth.login.config=/root/jaas.conf" Make Sure provide fully qualified path of the jaas.conf file. Create client.properties as per your cluster configurations. security.protocol=SASL_SSL
ssl.truststore.location=<truststore location>
ssl.truststore.password=<truststore password>
sasl.kerberos.service.name=kafka Use kafka-topics utility to list the topics in the cluster. kafka-topics --bootstrap-server <broker:port> --list --command-config client.properties
... View more
10-25-2024
05:10 AM
1 Kudo
From the below , it does looks like tez session itself not initialized. Validate configuration from set -v , make sure everything fine and try re-running the query. org.apache.tez.dag.api.SessionNotRunning: TezSession has already shutdown. Application application_1726980746968_0077 failed 1 times (global limit =2; local limit is =1) due to AM Container for appattempt_1726980746968_0077_000001 exited with exitCode: -1000
Failing this attempt.Diagnostics: [2024-10-13 10:30:27.706]Application application_1726980746968_0077 initialization failed (exitCode=255) with output: main : command provided 0 If you can't able to identify the incorrect configuration , raise a support case for the same.
... View more