hive merge statement and auto.convert.join

[#1]

Hello,

For serveral days I encountered an NPE error on hive when I execute several merge on the same table. When I execute the first merge on the table its ok but when i execute the second merge i get this error "Error: Error while compiling statement: FAILED: NullPointerException null (state=42000,code=40000)".

After many searches, i have try this in my code:

set hive.auto.convert.join=false;

and this resolved my NPE error but I don't understand why hive.auto.convert.join = true makes this error, knowing that this parameter checks if the smaller table file size is greater than the value specified by hive.mapjoin .smalltable.filesize.

Can someone help me to understand this strange behaviour ?

Thanks in advance

Best Regard

[#2]

Can you please provide the full trace ?

What do you mean by first merge and 2nd merge? COuld you please brief us ?

[#3]

Hi

The request i execute is something like this :

create table if not exists merge_mchb_mska_msku (col1 varchar(4), col2 varchar(8), col3 varchar(10) ...);

--first merge with table mchb
merge into sandbox.merge_mchb_mska f
using (col1, col2, col3 ...) from table sandbox.mchb m
on m.key=f.key
when matched then update set ...
when not matched then insert (col1, col2, col3 ...) values (m.col1, m.col2, m.col3 ...)

-- second merge with table mska
merge into sandbox.merge_mchb_mska f
using (col1, col2, col3 ...) from table sandbox.mska m
on m.key=f.key
when matched then update set ...
when not matched then insert (col1, col2, col3 ...) values (m.col1, m.col2, m.col3 ...)

 For the log, here is the log for the 2nd merge:

2020-06-30T09:46:08,723 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: parse.MergeSemanticAnalyzer (:()) - Generate an operator pipeline to autogather column stats for table factinventory in query FROM


2020-06-30T09:46:08,763 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: parse.CalcitePlanner (:()) - Get metadata for subqueries
2020-06-30T09:46:08,763 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: parse.CalcitePlanner (:()) - Get metadata for destination tables
2020-06-30T09:46:08,771 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ql.Context (:()) - New scratch dir is hdfs://lvdcluster/tmp/hive/hive/16ba3bf1-1774-4bcd-b0a0-1d122df23311/hive_2020-06-30_09-46-08_723_909451561220924551-4131
2020-06-30T09:46:08,775 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: common.FileUtils (FileUtils.java:mkdir(580)) - Creating directory if it doesn't exist: hdfs://lvdcluster/tmp/hive/hive/16ba3bf1-1774-4bcd-b0a0-1d122df23311/hive_2020-06-30_09-46-08_723_909451561220924551-4131/-mr-10000/.hive-staging_hive_2020-06-30_09-46-08_723_909451561220924551-4131
2020-06-30T09:46:08,783 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: SessionState (:()) - unix_timestamp(void) is deprecated. Use current_timestamp instead.
2020-06-30T09:46:08,792 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: common.FileUtils (FileUtils.java:mkdir(580)) - Creating directory if it doesn't exist: hdfs://lvdcluster/tmp/hive/hive/16ba3bf1-1774-4bcd-b0a0-1d122df23311/_tmp_space.db/merge_tmp_table/.hive-staging_hive_2020-06-30_09-46-08_645_758476176990790429-4131
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FS(20)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FS(27)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(26)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for GBY(25)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(24)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for GBY(23)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(22)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(19)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(18)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(17)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FS(35)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(34)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(33)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(32)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(31)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(30)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(29)
2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(28)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FS(43)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(42)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(41)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for GBY(40)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(39)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for GBY(38)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(37)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(36)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(45)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(47)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for JOIN(16)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(15)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(12)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(11)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(10)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(9)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for PTF(8)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(7)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(6)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(5)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(4)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for PTF(3)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(2)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(1)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for TS(0)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(14)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(44)
2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for TS(13)
2020-06-30T09:46:08,812 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 32 oldColExprMap: {VALUE._col30=Column[_col30], VALUE._col31=Column[_col31], VALUE._col10=Column[_col10], VALUE._col32=Const string 2020-06-30 09:46:08, VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col24=Column[_col24], VALUE._col25=Column[_col25], VALUE._col26=Column[_col26], VALUE._col27=Column[_col27], VALUE._col28=Column[_col28], VALUE._col29=Const string u, VALUE._col20=Column[_col20], VALUE._col21=Column[_col21], VALUE._col22=Column[_col22], VALUE._col23=Column[_col23], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]}
2020-06-30T09:46:08,812 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 32 newColExprMap: {VALUE._col30=Column[_col30], VALUE._col31=Column[_col31], VALUE._col10=Column[_col10], VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col24=Column[_col24], VALUE._col25=Column[_col25], VALUE._col26=Column[_col26], VALUE._col27=Column[_col27], VALUE._col28=Column[_col28], VALUE._col20=Column[_col20], VALUE._col21=Column[_col21], VALUE._col22=Column[_col22], VALUE._col23=Column[_col23], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]}
2020-06-30T09:46:08,812 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 24 oldColExprMap: {VALUE._col30=Column[_col30], VALUE._col31=Column[_col31], VALUE._col10=Column[_col10], VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col24=Column[_col24], VALUE._col25=Column[_col25], VALUE._col26=Column[_col26], VALUE._col27=Column[_col27], VALUE._col28=Column[_col28], VALUE._col29=Column[_col29], VALUE._col20=Column[_col20], VALUE._col21=Column[_col21], VALUE._col22=Column[_col22], VALUE._col23=Column[_col23], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]}
2020-06-30T09:46:08,812 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 24 newColExprMap: {VALUE._col30=Column[_col30], VALUE._col31=Column[_col31], VALUE._col10=Column[_col10], VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col24=Column[_col24], VALUE._col25=Column[_col25], VALUE._col26=Column[_col26], VALUE._col27=Column[_col27], VALUE._col28=Column[_col28], VALUE._col29=Column[_col29], VALUE._col20=Column[_col20], VALUE._col21=Column[_col21], VALUE._col22=Column[_col22], VALUE._col23=Column[_col23], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]}
2020-06-30T09:46:08,812 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 32 oldColExprMap: {VALUE._col30=Column[_col30], VALUE._col31=Column[_col31], VALUE._col10=Column[_col10], VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col24=Column[_col24], VALUE._col25=Column[_col25], VALUE._col26=Column[_col26], VALUE._col27=Column[_col27], VALUE._col28=Column[_col28], VALUE._col20=Column[_col20], VALUE._col21=Column[_col21], VALUE._col22=Column[_col22], VALUE._col23=Column[_col23], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 32 newColExprMap: {VALUE._col30=Column[_col30], VALUE._col31=Column[_col31], VALUE._col10=Column[_col10], VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col24=Column[_col24], VALUE._col25=Column[_col25], VALUE._col26=Column[_col26], VALUE._col27=Column[_col27], VALUE._col28=Column[_col28], VALUE._col20=Column[_col20], VALUE._col21=Column[_col21], VALUE._col22=Column[_col22], VALUE._col23=Column[_col23], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 39 oldColExprMap: {KEY._col0=Column[_col0], VALUE._col0=Column[_col1]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 39 newColExprMap: {KEY._col0=Column[_col0], VALUE._col0=Column[_col1]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 30 oldColExprMap: {VALUE._col30=Const string 2020-06-30 09:46:08, VALUE._col10=Column[_col11], VALUE._col11=Column[_col12], VALUE._col12=Column[_col13], VALUE._col24=Column[_col25], VALUE._col25=Column[_col26], VALUE._col26=Column[_col27], VALUE._col27=Column[_col28], VALUE._col28=Column[_col30], VALUE._col29=Column[_col31], VALUE._col20=Column[_col21], VALUE._col21=Column[_col22], VALUE._col22=Column[_col23], VALUE._col23=Column[_col24], KEY.reducesinkkey0=Column[_col0], VALUE._col2=Column[_col3], VALUE._col3=Column[_col4], VALUE._col4=Column[_col5], VALUE._col5=Column[_col6], VALUE._col0=Column[_col1], VALUE._col1=Column[_col2], VALUE._col13=Column[_col14], VALUE._col14=Column[_col15], VALUE._col15=Column[_col16], VALUE._col16=Column[_col17], VALUE._col6=Column[_col7], VALUE._col17=Column[_col18], VALUE._col7=Column[_col8], VALUE._col18=Column[_col19], VALUE._col8=Column[_col9], VALUE._col19=Column[_col20], VALUE._col9=Column[_col10]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 30 newColExprMap: {VALUE._col10=Column[_col11], VALUE._col11=Column[_col12], VALUE._col12=Column[_col13], VALUE._col24=Column[_col25], VALUE._col25=Column[_col26], VALUE._col26=Column[_col27], VALUE._col27=Column[_col28], VALUE._col28=Column[_col30], VALUE._col29=Column[_col31], VALUE._col20=Column[_col21], VALUE._col21=Column[_col22], VALUE._col22=Column[_col23], VALUE._col23=Column[_col24], KEY.reducesinkkey0=Column[_col0], VALUE._col2=Column[_col3], VALUE._col3=Column[_col4], VALUE._col4=Column[_col5], VALUE._col5=Column[_col6], VALUE._col0=Column[_col1], VALUE._col1=Column[_col2], VALUE._col13=Column[_col14], VALUE._col14=Column[_col15], VALUE._col15=Column[_col16], VALUE._col16=Column[_col17], VALUE._col6=Column[_col7], VALUE._col17=Column[_col18], VALUE._col7=Column[_col8], VALUE._col18=Column[_col19], VALUE._col8=Column[_col9], VALUE._col19=Column[_col20], VALUE._col9=Column[_col10]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 30 oldColExprMap: {VALUE._col10=Column[_col11], VALUE._col11=Column[_col12], VALUE._col12=Column[_col13], VALUE._col24=Column[_col25], VALUE._col25=Column[_col26], VALUE._col26=Column[_col27], VALUE._col27=Column[_col28], VALUE._col28=Column[_col30], VALUE._col29=Column[_col31], VALUE._col20=Column[_col21], VALUE._col21=Column[_col22], VALUE._col22=Column[_col23], VALUE._col23=Column[_col24], KEY.reducesinkkey0=Column[_col0], VALUE._col2=Column[_col3], VALUE._col3=Column[_col4], VALUE._col4=Column[_col5], VALUE._col5=Column[_col6], VALUE._col0=Column[_col1], VALUE._col1=Column[_col2], VALUE._col13=Column[_col14], VALUE._col14=Column[_col15], VALUE._col15=Column[_col16], VALUE._col16=Column[_col17], VALUE._col6=Column[_col7], VALUE._col17=Column[_col18], VALUE._col7=Column[_col8], VALUE._col18=Column[_col19], VALUE._col8=Column[_col9], VALUE._col19=Column[_col20], VALUE._col9=Column[_col10]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 30 newColExprMap: {VALUE._col10=Column[_col11], VALUE._col11=Column[_col12], VALUE._col12=Column[_col13], VALUE._col24=Column[_col25], VALUE._col25=Column[_col26], VALUE._col26=Column[_col27], VALUE._col27=Column[_col28], VALUE._col28=Column[_col30], VALUE._col29=Column[_col31], VALUE._col20=Column[_col21], VALUE._col21=Column[_col22], VALUE._col22=Column[_col23], VALUE._col23=Column[_col24], KEY.reducesinkkey0=Column[_col0], VALUE._col2=Column[_col3], VALUE._col3=Column[_col4], VALUE._col4=Column[_col5], VALUE._col5=Column[_col6], VALUE._col0=Column[_col1], VALUE._col1=Column[_col2], VALUE._col13=Column[_col14], VALUE._col14=Column[_col15], VALUE._col15=Column[_col16], VALUE._col16=Column[_col17], VALUE._col6=Column[_col7], VALUE._col17=Column[_col18], VALUE._col7=Column[_col8], VALUE._col18=Column[_col19], VALUE._col8=Column[_col9], VALUE._col19=Column[_col20], VALUE._col9=Column[_col10]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - JOIN 16 oldExprs: {0=[Column[KEY.reducesinkkey0], Column[VALUE._col0], Column[KEY.reducesinkkey2], Column[KEY.reducesinkkey3], Column[KEY.reducesinkkey4], Column[KEY.reducesinkkey5], Column[VALUE._col1], Column[VALUE._col2], Column[VALUE._col3], Column[VALUE._col4], Column[VALUE._col5], Column[VALUE._col6], Column[VALUE._col7], Column[VALUE._col8], Column[VALUE._col9], Column[VALUE._col10], Column[VALUE._col11], Column[VALUE._col12], Column[VALUE._col13], Column[VALUE._col14], Column[VALUE._col15], Column[VALUE._col16], Column[VALUE._col17], Column[VALUE._col18], Column[VALUE._col19], Column[VALUE._col20], Column[VALUE._col21], Column[VALUE._col22], Column[VALUE._col23], Column[VALUE._col24], Column[VALUE._col25], Column[VALUE._col26], Column[VALUE._col27], Column[VALUE._col28], Column[VALUE._col29]], 1=[Column[KEY.reducesinkkey0], Column[KEY.reducesinkkey1], Column[KEY.reducesinkkey2], Column[KEY.reducesinkkey3], Column[KEY.reducesinkkey4], Column[KEY.reducesinkkey5], Column[KEY.reducesinkkey6], Column[VALUE._col0], Column[VALUE._col1], Column[VALUE._col2], Column[VALUE._col3], Column[VALUE._col4], Column[VALUE._col5], Column[VALUE._col6], Column[VALUE._col7], Column[VALUE._col8], Column[VALUE._col9], Column[VALUE._col10], Column[VALUE._col11], Column[VALUE._col12], Column[VALUE._col13]]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - JOIN 16 newExprs: {0=[Column[KEY.reducesinkkey0], Column[VALUE._col0], Column[KEY.reducesinkkey2], Column[KEY.reducesinkkey3], Column[KEY.reducesinkkey4], Column[KEY.reducesinkkey5], Column[VALUE._col1], Column[VALUE._col2], Column[VALUE._col3], Column[VALUE._col5], Column[VALUE._col6], Column[VALUE._col13], Column[VALUE._col14], Column[VALUE._col15], Column[VALUE._col19], Column[VALUE._col20], Column[VALUE._col21], Column[VALUE._col29]], 1=[Column[KEY.reducesinkkey0], Column[KEY.reducesinkkey1], Column[KEY.reducesinkkey2], Column[KEY.reducesinkkey3], Column[KEY.reducesinkkey4], Column[KEY.reducesinkkey5], Column[KEY.reducesinkkey6], Column[VALUE._col0], Column[VALUE._col1], Column[VALUE._col2], Column[VALUE._col3], Column[VALUE._col4], Column[VALUE._col5], Column[VALUE._col6], Column[VALUE._col7], Column[VALUE._col8], Column[VALUE._col9], Column[VALUE._col10], Column[VALUE._col11], Column[VALUE._col12], Column[VALUE._col13]]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 14 oldColExprMap: {VALUE._col10=Column[unrestrictedstock], VALUE._col11=Column[qualitystock], VALUE._col12=Column[blockedstock], VALUE._col24=Column[odq_changemode], VALUE._col25=Column[timestmp], VALUE._col26=Column[tdate], VALUE._col27=Column[BLOCK__OFFSET__INSIDE__FILE], VALUE._col28=Column[INPUT__FILE__NAME], VALUE._col29=Column[ROW__ID], VALUE._col20=Column[previousstockintransfer], VALUE._col21=Column[previousrestrictedstock], VALUE._col22=Column[origin], VALUE._col23=Column[udp], KEY.reducesinkkey0=Column[plant], KEY.reducesinkkey1=Column[product], KEY.reducesinkkey2=Column[batch], KEY.reducesinkkey3=Column[storagelocation], KEY.reducesinkkey4=Column[specialstockindicator], KEY.reducesinkkey5=Column[salesdocument], KEY.reducesinkkey6=Column[salesdocumentitem], VALUE._col2=Column[customer], VALUE._col3=Column[vendor], VALUE._col4=Column[creationdate], VALUE._col5=Column[lastchangedate], VALUE._col0=Column[product], VALUE._col1=Column[salesdocumentitem], VALUE._col13=Column[returns], VALUE._col14=Column[stockintransfer], VALUE._col15=Column[restrictedstock], VALUE._col16=Column[previousunrestrictedstock], VALUE._col6=Column[deletionflag], VALUE._col17=Column[previousqualitystock], VALUE._col7=Column[blockedinventoryindicator], VALUE._col18=Column[previousblockedstock], VALUE._col8=Column[currentyear], VALUE._col19=Column[previousreturns], VALUE._col9=Column[currentperiod]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 14 newColExprMap: {VALUE._col29=Column[ROW__ID], VALUE._col20=Column[previousstockintransfer], VALUE._col21=Column[previousrestrictedstock], KEY.reducesinkkey0=Column[plant], KEY.reducesinkkey1=Column[product], KEY.reducesinkkey2=Column[batch], KEY.reducesinkkey3=Column[storagelocation], KEY.reducesinkkey4=Column[specialstockindicator], KEY.reducesinkkey5=Column[salesdocument], KEY.reducesinkkey6=Column[salesdocumentitem], VALUE._col2=Column[customer], VALUE._col3=Column[vendor], VALUE._col5=Column[lastchangedate], VALUE._col0=Column[product], VALUE._col1=Column[salesdocumentitem], VALUE._col13=Column[returns], VALUE._col14=Column[stockintransfer], VALUE._col15=Column[restrictedstock], VALUE._col6=Column[deletionflag], VALUE._col19=Column[previousreturns]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 15 oldColExprMap: {VALUE._col10=Column[_col17], VALUE._col11=Column[_col18], VALUE._col12=Column[_col19], KEY.reducesinkkey0=Column[_col0], KEY.reducesinkkey1=Column[_col1], KEY.reducesinkkey2=Column[_col2], KEY.reducesinkkey3=Column[_col3], KEY.reducesinkkey4=Column[_col4], KEY.reducesinkkey5=Column[_col5], KEY.reducesinkkey6=Column[_col6], VALUE._col2=Column[_col9], VALUE._col3=Column[_col10], VALUE._col4=Column[_col11], VALUE._col5=Column[_col12], VALUE._col0=Column[_col7], VALUE._col1=Column[_col8], VALUE._col13=Column[_col20], VALUE._col6=Column[_col13], VALUE._col7=Column[_col14], VALUE._col8=Column[_col15], VALUE._col9=Column[_col16]}
2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 15 newColExprMap: {VALUE._col10=Column[_col17], VALUE._col11=Column[_col18], VALUE._col12=Column[_col19], KEY.reducesinkkey0=Column[_col0], KEY.reducesinkkey1=Column[_col1], KEY.reducesinkkey2=Column[_col2], KEY.reducesinkkey3=Column[_col3], KEY.reducesinkkey4=Column[_col4], KEY.reducesinkkey5=Column[_col5], KEY.reducesinkkey6=Column[_col6], VALUE._col2=Column[_col9], VALUE._col3=Column[_col10], VALUE._col4=Column[_col11], VALUE._col5=Column[_col12], VALUE._col0=Column[_col7], VALUE._col1=Column[_col8], VALUE._col13=Column[_col20], VALUE._col6=Column[_col13], VALUE._col7=Column[_col14], VALUE._col8=Column[_col15], VALUE._col9=Column[_col16]}
2020-06-30T09:46:08,814 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 6 oldColExprMap: {VALUE._col10=Column[_col17], VALUE._col11=Column[_col18], VALUE._col12=Column[_col19], KEY.reducesinkkey0=Column[_col2], KEY.reducesinkkey1=Column[_col3], KEY.reducesinkkey2=Column[_col1], KEY.reducesinkkey3=Column[_col6], KEY.reducesinkkey4=Column[_col4], KEY.reducesinkkey5=Column[_col5], KEY.reducesinkkey6=Column[_col0], KEY.reducesinkkey7=Column[_col20], VALUE._col2=Column[_col9], KEY.reducesinkkey8=Column[_col21], VALUE._col3=Column[_col10], VALUE._col4=Column[_col11], VALUE._col5=Column[_col12], VALUE._col0=Column[_col7], VALUE._col1=Column[_col8], VALUE._col6=Column[_col13], VALUE._col7=Column[_col14], VALUE._col8=Column[_col15], VALUE._col9=Column[_col16]}
2020-06-30T09:46:08,814 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 6 newColExprMap: {VALUE._col10=Column[_col17], VALUE._col11=Column[_col18], VALUE._col12=Column[_col19], KEY.reducesinkkey0=Column[_col2], KEY.reducesinkkey1=Column[_col3], KEY.reducesinkkey2=Column[_col1], KEY.reducesinkkey3=Column[_col6], KEY.reducesinkkey4=Column[_col4], KEY.reducesinkkey5=Column[_col5], KEY.reducesinkkey6=Column[_col0], KEY.reducesinkkey7=Column[_col20], VALUE._col2=Column[_col9], KEY.reducesinkkey8=Column[_col21], VALUE._col3=Column[_col10], VALUE._col4=Column[_col11], VALUE._col5=Column[_col12], VALUE._col0=Column[_col7], VALUE._col1=Column[_col8], VALUE._col6=Column[_col13], VALUE._col7=Column[_col14], VALUE._col8=Column[_col15], VALUE._col9=Column[_col16]}
2020-06-30T09:46:08,814 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 1 oldColExprMap: {VALUE._col10=Column[lfmon], VALUE._col11=Column[kalab], VALUE._col12=Column[kains], VALUE._col20=Column[date_migration], VALUE._col21=Column[BLOCK__OFFSET__INSIDE__FILE], VALUE._col22=Column[INPUT__FILE__NAME], VALUE._col23=Column[ROW__ID], KEY.reducesinkkey0=Const int 0, VALUE._col2=Column[charg], VALUE._col3=Column[lgort], VALUE._col4=Column[sobkz], VALUE._col5=Column[vbeln], VALUE._col0=Column[werks], VALUE._col1=Column[matnr], VALUE._col13=Column[kaspe], VALUE._col14=Column[kavla], VALUE._col15=Column[kavin], VALUE._col16=Column[kavsp], VALUE._col6=Column[posnr], VALUE._col17=Column[origin], VALUE._col7=Column[ersda], VALUE._col18=Column[udp], VALUE._col8=Column[kaspr], VALUE._col19=Column[odq_changemode], VALUE._col9=Column[lfgja]}
2020-06-30T09:46:08,814 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 1 newColExprMap: {VALUE._col10=Column[lfmon], VALUE._col11=Column[kalab], VALUE._col12=Column[kains], VALUE._col20=Column[date_migration], KEY.reducesinkkey0=Const int 0, VALUE._col2=Column[charg], VALUE._col3=Column[lgort], VALUE._col4=Column[sobkz], VALUE._col5=Column[vbeln], VALUE._col0=Column[werks], VALUE._col1=Column[matnr], VALUE._col13=Column[kaspe], VALUE._col14=Column[kavla], VALUE._col15=Column[kavin], VALUE._col16=Column[kavsp], VALUE._col6=Column[posnr], VALUE._col17=Column[origin], VALUE._col7=Column[ersda], VALUE._col18=Column[udp], VALUE._col8=Column[kaspr], VALUE._col19=Column[odq_changemode], VALUE._col9=Column[lfgja]}
2020-06-30T09:46:08,814 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: correlation.AbstractCorrelationProcCtx (:()) - Overriding hive.optimize.reducededuplication.min.reducer to 1 due to a write to transactional table(s) refining_silver_test.factinventory,refining_silver_test.factinventory
2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.SetReducerParallelism (:()) - Set parallelism for reduce sink RS[1] to: 16
2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.SetReducerParallelism (:()) - Set parallelism for reduce sink RS[14] to: 84
2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.SetReducerParallelism (:()) - Set parallelism for reduce sink RS[6] to: 16
2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.SetReducerParallelism (:()) - Set parallelism for reduce sink RS[15] to: 84
2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Memory monitor info set to : { isLlap: false executorsPerNode: 1 maxExecutorsOverSubscribeMemory: 3 memoryOverSubscriptionFactor: 0.20000000298023224 memoryCheckInterval: 100000 noConditionalTaskSize: 1717986918 adjustedNoConditionalTaskSize: 1717986918 hashTableInflationFactor: 2.0 threshold: 3435973836 }
2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - maxJoinMemory: 1717986918
2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Estimated number of buckets 1
2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Join input#0; onlineDataSize: 1870875047; Statistics: Num rows: 3287943 Data size: 5097628704 Basic stats: COMPLETE Column stats: NONE
2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Try dynamic partitioned hash join with estimated 84 reducers
2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Join input#0; onlineDataSize: 1870875047; Statistics: Num rows: 3287943 Data size: 5097628704 Basic stats: COMPLETE Column stats: NONE
2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Join input#1; onlineDataSize: 388737795; Statistics: Num rows: 133039 Data size: 522108828 Basic stats: COMPLETE Column stats: NONE
2020-06-30T09:46:08,841 ERROR [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ql.Driver (:()) - FAILED: NullPointerException null
java.lang.NullPointerException
at org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc.<init>(ExprNodeColumnDesc.java:84)
at org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc.<init>(ExprNodeColumnDesc.java:80)
at org.apache.hadoop.hive.ql.plan.ExprNodeDescUtils.resolveJoinKeysAsRSColumns(ExprNodeDescUtils.java:450)
at org.apache.hadoop.hive.ql.plan.ExprNodeDescUtils.resolveJoinKeysAsRSColumns(ExprNodeDescUtils.java:411)
at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.getMapJoinDesc(MapJoinProcessor.java:1103)
at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.convertJoinOpMapJoinOp(MapJoinProcessor.java:372)
at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.convertJoinMapJoin(ConvertJoinMapJoin.java:1194)
at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.convertJoinDynamicPartitionedHashJoin(ConvertJoinMapJoin.java:1417)
at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.fallbackToReduceSideJoin(ConvertJoinMapJoin.java:1449)
at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.checkAndConvertSMBJoin(ConvertJoinMapJoin.java:491)
at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.process(ConvertJoinMapJoin.java:158)
at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
at org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:74)
at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
at org.apache.hadoop.hive.ql.parse.TezCompiler.runStatsDependentOptimizations(TezCompiler.java:434)
at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:170)
at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12521)
at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11855)
at org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:66)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.parse.MergeSemanticAnalyzer.analyzeMerge(MergeSemanticAnalyzer.java:300)
at org.apache.hadoop.hive.ql.parse.MergeSemanticAnalyzer.analyze(MergeSemanticAnalyzer.java:62)
at org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:71)
at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1870)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1817)
at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1812)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197)
at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:262)
at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:575)
at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:561)
at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59)
at com.sun.proxy.$Proxy102.executeStatementAsync(Unknown Source)
at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315)
at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:566)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557)
at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

[#4]

I see that you are using LLAP:

2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Memory monitor info set to : { isLlap: false executorsPerNode: 1 maxExecutorsOverSubscribeMemory: 3 memoryOverSubscriptionFactor: 0.20000000298023224 memoryCheckInterval: 100000 noConditionalTaskSize: 1717986918 adjustedNoConditionalTaskSize: 1717986918 hashTableInflationFactor: 2.0 threshold: 3435973836 }
2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - maxJoinMemory: 1717986918

 The Map join in LLAP works differently  then traditional Hiveserver2. You can go through https://community.cloudera.com/t5/Community-Articles/Map-Join-Memory-Sizing-For-LLAP/ta-p/247462 for more information .

[#5]

Thank you @asish for answering,

it helps me understand a little better, but in my case, I'm not using Llap and the argument for isLlap is set to false. 

Is there anything else that can explain this behavior  (i think it is related to the set up of map join memory in hive config) ?

Best Regards

[#6]

I am not sure,I believe you are hitting https://issues.apache.org/jira/browse/HIVE-21799

This fails at compile phase .