question Re: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1) in Support Questions

Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1)

dmharshit — Mon, 26 Apr 2021 09:46:07 GMT

Error message:
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1)

Following is Details of our cluster

HDP Version: 3.1.4
Hive Version: 3.1.0.3.1.4.0-315

Configs for Dynamic Partition in hive:

hive.exec.dynamic.partition=true

Allow all partitions to be Dynamic=nonstrict

hive.exec.max.dynamic.partitions=50000

hive.exec.max.dynamic.partitions.pernode=20000

Sort Partitions Dynamically=false
Allow dynamic numbers of reducers=true
Allow dynamic partition pruning=true

Any help appreciated.

Re: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1)

shobikas — Tue, 27 Apr 2021 09:23:49 GMT

@dmharshit It would be really difficult to answer for the question without the necessary details like what is the query you are running, From when you started facing the issue and the complete error stack. But from the two line which you provided i suspect that there might be issue with stats. So you can try setting the below parameter and try it.

set hive.stats.column.autogather=false; 
set hive.optimize.sort.dynamic.partition=true;

If the above doesn't help kindly provide the necessary details to figure out the issue.

If the above parameter resolves the issue kindly mark "Accept as a Solution"

Thanks

Shobika S

Re: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1)

dmharshit — Wed, 28 Apr 2021 07:28:06 GMT

Hi @shobikas,

Thanks for your response.

I tried running queries using the configs you mentioned. Queries are getting executed successfully but no change is observed in table data.

Following queries I'm running in hive (for mix partitioning which includes static and dynamic partitioning together)
1. insert into table foodmart.idNameGenderPartDeptTableName partition (manager_id = 200, manager_name, manager_gender) VALUES (1, 'COMP', 'John', 'M');
2. insert into table foodmart.idNameGenderPartDeptTableName partition (manager_id = 1111, manager_name, manager_gender) select department_id, full_name, first_name, gender from foodmart.trimmed_employee;
3. insert OVERWRITE table foodmart.idNameGenderPartDeptTableName partition (manager_id = 1234, manager_name, manager_gender) select department_id, full_name, first_name, gender from foodmart.trimmed_employee;
4. insert OVERWRITE table foodmart.idNameGenderPartDeptTableName partition (manager_id = 1234, manager_name, manager_gender) values (02, 'HR', 'Niks', 'M');

Re: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1)

vidanimegh — Thu, 29 Apr 2021 05:55:00 GMT

Hi @dmharshit ,

Can you share the log snippets from Hive Metastore, Hiveserver2 and YARN ResourceManager from the timeframe of execution of this query?

Thanks,

Megh

Re: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1)

dmharshit — Wed, 09 Jun 2021 11:30:46 GMT

I compared the hive set ; between HIVE 2.x & HIVE 3.x

In HIVE 3.x following value is 'nonstrict'. Is that causing the issue?
hive.exec.dynamic.partition.mode=nonstrict

The problem is, if I execute the query which contains static & dynamic parameters I am getting the error.

If I just keep it static or dynamic parameters, no problem.

Some more parameters which might be causing the issue but I am not too sure =

hive.optimize.dynamic.partition.hashjoin=true hive.optimize.sort.dynamic.partition=false hive.server2.support.dynamic.service.discovery=true

Re: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1)

dmharshit — Fri, 11 Jun 2021 07:32:40 GMT

@shobikas @vidanimegh Pls suggest thanks.

Re: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1)

dmharshit — Mon, 14 Jun 2021 04:20:56 GMT

@Shelton Any hints on this issue? Thanks.

Re: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1)

vidanimegh — Mon, 14 Jun 2021 06:50:01 GMT

Hi @dmharshit ,

It is difficult to comment without looking at the logs.

Kindly share the log snippets from HiveServer2, Hive Metastore and YARN during the execution of this query.

Thanks,

Megh

Re: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1)

dmharshit — Mon, 14 Jun 2021 07:24:37 GMT

I tried with the logs but unable to find any traces with the keywords.
I even tried to check the logs by killing all jobs.

Re: Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.StatsTask (state=08S01,code=1)

vidanimegh — Fri, 18 Jun 2021 06:38:56 GMT

@dmharshit ,

When you run this query, does a YARN application ID get generated? Or the query fails before triggering the YARN application?

In case YARN application is triggered, please get the logs of that particular YARN application and check for errors.

yarn logs -applicationId your_application_id > your_application_id.log 2>&1

Check and see if you're able to get any detailed errors in this log file and share.

Thanks,

Megh