Member since
10-28-2020
554
Posts
45
Kudos Received
39
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3787 | 07-23-2024 11:49 PM | |
542 | 05-28-2024 11:06 AM | |
972 | 05-05-2024 01:27 PM | |
633 | 05-05-2024 01:09 PM | |
650 | 03-28-2024 09:51 AM |
03-20-2024
03:54 AM
2 Kudos
@Choolake See if this does the job for you. ...
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.JsonSerde'
WITH SERDEPROPERTIES (
'separatorChar' = '|',
'quoteChar' = '"'
)
STORED AS TEXTFILE LOCATION .... This is a third party serde. You may download it from https://code.google.com/archive/p/hive-json-serde/downloads
... View more
03-20-2024
02:59 AM
1 Kudo
@ZainK This can happen due to various reasons such as resource constraints, or errors in the code. Do check those aspects.
... View more
03-17-2024
09:40 AM
@Hadoop16 Was it working before? Did anything change from Kerberos point of view? Try regenerating the hive keytab file and see if it helps.
... View more
03-14-2024
11:58 PM
1 Kudo
Hive does use stats from an external table in preparing query plan. When stats are accurate, it could estimate the size of intermediate data sets and select efficient join strategies. The only thing I noticed is the fetch task is not working.
... View more
03-14-2024
06:29 AM
Try this: CREATE external TABLE mytable (
col1 INT,
col2 STRING,
col3 STRING,
col4 STRING,
col5 INT,
col6 STRING,
col7 STRING
...
)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
WITH SERDEPROPERTIES (
"separatorChar" = ",",
"quoteChar" = "\""
)
STORED AS TEXTFILE; It should work.
... View more
03-12-2024
11:56 PM
1 Kudo
@yashwanth It seems like you want to separate columns based on the position of comma. In that case, you may create the table as follows: CREATE TABLE my_table (
col1 STRING,
col2 INT,
...
)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ',' ...
... View more
03-12-2024
07:46 AM
@Leopold It is disabled for external tables as data in HDFS can change without Hive knowing about it. Unfortunately I do not see a way to enforce fetch task for a query with an aggregate function.
... View more
03-12-2024
04:00 AM
1 Kudo
@Leopold I just checked. Your observation is correct. For external tables, it does not use a fetch task. In the logs, I see the following message: 2024-03-12 10:48:37,247 INFO org.apache.hadoop.hive.ql.optimizer.StatsOptimizer: [b226e7aa-9a42-4af3-b99b-be4a6592fb7f HiveServer2-Handler-Pool: Thread-31145]: Table t7 is external. Skip StatsOptimizer. But enabling "hive.fetch.task.aggr=true" will help avoid the Reducer phase that is used for final aggregation. It will be a Map-only job.
... View more
03-12-2024
02:38 AM
@Leopold provided we have column stats available, Hive could use a fetch task to perform a simple aggregation task such as max(), instead of launching a Map job. Try hive.fetch.task.aggr=true . This property is disabled by default.
... View more
02-26-2024
05:21 AM
@frbelotto It does not require any installation. You just need to share the path to the Driver jar file.
... View more