Member since
11-11-2019
634
Posts
33
Kudos Received
27
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 260 | 10-09-2025 12:29 AM | |
| 4771 | 02-19-2025 09:43 PM | |
| 2124 | 02-28-2023 09:32 PM | |
| 4002 | 02-27-2023 03:33 AM | |
| 26008 | 12-24-2022 05:56 AM |
08-24-2021
02:06 AM
Is hive generating slow response? Is the query slow??
... View more
08-03-2021
03:36 AM
DOes the same issue happen when you run via beeline ? I researched ,I found this occurs only in hue.
... View more
08-03-2021
03:25 AM
Hi Megh,
Thanks for the logs.
I found a discrepancy in HDP logs,as per "Query counters on HDP Cluster.txt" ,the Mapper spawned is 64 but from the application_1627033595344_12159_hdp.log,I observe it is 127.
VERTICES MODE STATUS TOTAL COMPLETED RUNNING PENDING FAILED KILLED
----------------------------------------------------------------------------------------------
Map 1 .......... container SUCCEEDED 64 64 0 0 0 0
Reducer 2 ...... container SUCCEEDED 1 1 0 0 0 0
----------------------------------------------------------------------------------------------
grep "Event:TASK_STARTED]: vertexName=Map 1" syslog_dag_1627033595344_12159_1 | wc -l
127
grep "Event:TASK_FINISHED]: vertexName=Map 1" syslog_dag_1627033595344_12159_1 | wc -l
127
Question: have you provided correct application log for HDP ? As I see a discrepancy.
Assuming you have provided correct set of logs for HDO,I compared between CDP and HDP,I see that input records for each maper is more in CDP then HDP that is the reason it took more tie in CDP.
I have collected samle taskID and INPUT_RECORDS_PROCESSED for each maper.
application_1627885418606_0016_cdp.log
======
each Mapper took close to 1 min to finish
taskId=task_1627885418606_0016_1_00_000059 INPUT_RECORDS_PROCESSED=2012469
taskId=task_1627885418606_0016_1_00_000061 INPUT_RECORDS_PROCESSED=2010369
taskId=task_1627885418606_0016_1_00_000026 INPUT_RECORDS_PROCESSED=2021105
application_1627033595344_12159_hdp.log
==========
taskId=task_1627033595344_12159_1_00_000062 INPUT_RECORDS_PROCESSED=1010250
taskId=task_1627033595344_12159_1_00_000126 INPUT_RECORDS_PROCESSED=376062
Thanks,
Asish
... View more
07-25-2021
07:56 AM
I would need complete application logs,HS2 logs and beeline trace to analyse. The snippet provided does not give much info.
... View more
07-20-2021
01:11 AM
Are you checking from beeline/any other JDBC client ? You are seeing this because your window is not properly formatted.
... View more
07-20-2021
01:06 AM
Can you check "show locks" if the table is acquired EXCLUSIVE lock ?
... View more
07-19-2021
10:26 AM
Hi Sai, Please follow https://community.cloudera.com/t5/Support-Questions/Does-hive-support-Photo-or-images-datatypes/td-p/221473 Copy the data to hdfs instead of loadpath and check, Thanks, Asish
... View more
07-18-2021
08:37 AM
All the hive related tables are stored under "hive" database in mysql. You can take mysql dump for a database hive and can prevent this from happening in the future. You can use command like: mysqldump -u root -p hive Reference: https://www.sqlshack.com/how-to-backup-and-restore-mysql-databases-using-the-mysqldump-command/
... View more
07-18-2021
08:32 AM
Following is the criteria:
ORC and Managed table
ACID enabled
HDFS directory should be owned by Hive
According to the shouldTableBeExternal method in HiveStrictMigration.java, a table is external if it is a StorageHandler table, or if it is an Avro/Text/Parquet table, or if it is a list bucketed table, or the directory is not owned by Hive.
Following is the codebase that describes the same:
String reason = shouldTableBeExternal(tableObj, ownerName, conf, hms, isPathOwnedByHive); if (reason != null) { LOG.debug("Converting {} to external table. {}", getQualifiedName(tableObj), reason); result = TableMigrationOption.EXTERNAL; } else { result = TableMigrationOption.MANAGED;
Reference: HiveStrictManagedMigration (GitHub)
It should be converted to an EXTERNAL table, in case it does not fit the criteria for the upgrade.
... View more
07-18-2021
08:22 AM
Please provide the below information: 1. What is the version? HDP/CDP/CDH ? 2. provide "scriptpro" 3. What si the error recieved ?
... View more