question Impala can not read the files stored in the hive dir in hdfs in Support Questions

Impala can not read the files stored in the hive dir in hdfs

drgenious — Sat, 07 Aug 2021 11:09:50 GMT

Hi,

I have a table in Impala and it had some files in its location in hdfs. All of a sudden it can not read the files. I tried to invalidate and recover the partitions but nothing happened. I also moved the files from this folder into a new's folder path but it couldn't read them too.

What can I do ?

UPDATE

Tried to repair the table in hive

set hive.msck.repair.batch.size=1;
set hive.msck.path.validation=ignore;
msck REPAIR TABLE db.table

but received this error :

Error while processing statement: FAILED: Execution Error, return code 1 
from org.apache.hadoop.hive.ql.exec.DDLTask

Re: Impala can not read the files stored in the hive dir in hdfs

ChethanYM — Sun, 08 Aug 2021 16:47:28 GMT

Hi,

Can you provide these below details to understand the issue:

1. Initially from where you created the table in hive or impala?

2. Provide the DDL of the table that you are trying to query from impala.

# show create table <table-name>;

3. list the files from hdfs path ( you will see the table path in the DDL def. )

# hdfs dfs -ls /table/path/

4. Are you able to access the same table data from hive?( if you are facing the issues from impala? vice versa)

5. Go to HS2 logs/coordinator logs copy the entire error stack trace for the query that is failing and paste here.

I think these are the basic details needed for analysing the issue.

Regards,

Chethan YM

Re: Impala can not read the files stored in the hive dir in hdfs

drgenious — Mon, 09 Aug 2021 06:14:23 GMT

Hi @ChethanYM ,

1)I have created in impala.

2)result
CREATE TABLE db.table (
...
)
PARTITIONED BY (
...
)
SORT BY (
...
)
STORED AS PARQUET
LOCATION 'hdfs://path/'
TBLPROPERTIES ('DO_NOT_UPDATE_STATS'='true', 'STATS_GENERATED'='TASK', 'impala.lastComputeStatsTime'='1627655409', 'numRows'='0', 'parquet.compression'='snappy', 'totalSize'='0')

3) I have listed the files and they seem ok

4)I can access it but returns again 0 results

5)It is not failing the query. It can not read the partitions as it did before.

Re: Impala can not read the files stored in the hive dir in hdfs

ChethanYM — Tue, 10 Aug 2021 00:40:40 GMT

Hi,

What is the query you are using to read the data from table? can you attach its "query profile" and coordinator logs to have a look?

Regards,

Chethan YM