Member since
12-02-2019
19
Posts
4
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1833 | 04-08-2020 06:13 AM | |
| 1718 | 03-06-2020 08:28 AM |
09-15-2021
05:05 AM
Hello,
For an application, I need to extract the maximum depth from an hdfs directory. I know how to do this action in shell: we can execute
find /tmp -type d -printf '%d\n' | sort -rn | head -1
So I wanted to do the same with the find function of hdfs:
hdfs dfs -find /tmp -type d
but the -type argument does not exist on hdfs, here is the error:
find: Unexpected argument: -type
Does anyone have any solution or advice for this problem ?
ps: my hadoop version Hadoop 2.6.0-cdh5.13.
regards,
thanks in advance
... View more
Labels:
- Labels:
-
Apache Sqoop
-
HDFS
07-08-2020
07:16 AM
Thank you @asish for answering, it helps me understand a little better, but in my case, I'm not using Llap and the argument for isLlap is set to false. Is there anything else that can explain this behavior (i think it is related to the set up of map join memory in hive config) ? Best Regards
... View more
07-02-2020
03:00 AM
Hi The request i execute is something like this : create table if not exists merge_mchb_mska_msku (col1 varchar(4), col2 varchar(8), col3 varchar(10) ...); --first merge with table mchb merge into sandbox.merge_mchb_mska f using (col1, col2, col3 ...) from table sandbox.mchb m on m.key=f.key when matched then update set ... when not matched then insert (col1, col2, col3 ...) values (m.col1, m.col2, m.col3 ...) -- second merge with table mska merge into sandbox.merge_mchb_mska f using (col1, col2, col3 ...) from table sandbox.mska m on m.key=f.key when matched then update set ... when not matched then insert (col1, col2, col3 ...) values (m.col1, m.col2, m.col3 ...) For the log, here is the log for the 2nd merge: 2020-06-30T09:46:08,723 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: parse.MergeSemanticAnalyzer (:()) - Generate an operator pipeline to autogather column stats for table factinventory in query FROM 2020-06-30T09:46:08,763 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: parse.CalcitePlanner (:()) - Get metadata for subqueries 2020-06-30T09:46:08,763 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: parse.CalcitePlanner (:()) - Get metadata for destination tables 2020-06-30T09:46:08,771 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ql.Context (:()) - New scratch dir is hdfs://lvdcluster/tmp/hive/hive/16ba3bf1-1774-4bcd-b0a0-1d122df23311/hive_2020-06-30_09-46-08_723_909451561220924551-4131 2020-06-30T09:46:08,775 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: common.FileUtils (FileUtils.java:mkdir(580)) - Creating directory if it doesn't exist: hdfs://lvdcluster/tmp/hive/hive/16ba3bf1-1774-4bcd-b0a0-1d122df23311/hive_2020-06-30_09-46-08_723_909451561220924551-4131/-mr-10000/.hive-staging_hive_2020-06-30_09-46-08_723_909451561220924551-4131 2020-06-30T09:46:08,783 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: SessionState (:()) - unix_timestamp(void) is deprecated. Use current_timestamp instead. 2020-06-30T09:46:08,792 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: common.FileUtils (FileUtils.java:mkdir(580)) - Creating directory if it doesn't exist: hdfs://lvdcluster/tmp/hive/hive/16ba3bf1-1774-4bcd-b0a0-1d122df23311/_tmp_space.db/merge_tmp_table/.hive-staging_hive_2020-06-30_09-46-08_645_758476176990790429-4131 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FS(20) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FS(27) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(26) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for GBY(25) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(24) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for GBY(23) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(22) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(19) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(18) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(17) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FS(35) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(34) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(33) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(32) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(31) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(30) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(29) 2020-06-30T09:46:08,807 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(28) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FS(43) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(42) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(41) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for GBY(40) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(39) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for GBY(38) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(37) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(36) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(45) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(47) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for JOIN(16) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(15) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(12) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(11) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(10) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(9) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for PTF(8) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(7) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(6) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(5) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(4) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for PTF(3) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for SEL(2) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(1) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for TS(0) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for RS(14) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for FIL(44) 2020-06-30T09:46:08,808 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ppd.OpProcFactory (:()) - Processing for TS(13) 2020-06-30T09:46:08,812 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 32 oldColExprMap: {VALUE._col30=Column[_col30], VALUE._col31=Column[_col31], VALUE._col10=Column[_col10], VALUE._col32=Const string 2020-06-30 09:46:08, VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col24=Column[_col24], VALUE._col25=Column[_col25], VALUE._col26=Column[_col26], VALUE._col27=Column[_col27], VALUE._col28=Column[_col28], VALUE._col29=Const string u, VALUE._col20=Column[_col20], VALUE._col21=Column[_col21], VALUE._col22=Column[_col22], VALUE._col23=Column[_col23], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]} 2020-06-30T09:46:08,812 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 32 newColExprMap: {VALUE._col30=Column[_col30], VALUE._col31=Column[_col31], VALUE._col10=Column[_col10], VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col24=Column[_col24], VALUE._col25=Column[_col25], VALUE._col26=Column[_col26], VALUE._col27=Column[_col27], VALUE._col28=Column[_col28], VALUE._col20=Column[_col20], VALUE._col21=Column[_col21], VALUE._col22=Column[_col22], VALUE._col23=Column[_col23], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]} 2020-06-30T09:46:08,812 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 24 oldColExprMap: {VALUE._col30=Column[_col30], VALUE._col31=Column[_col31], VALUE._col10=Column[_col10], VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col24=Column[_col24], VALUE._col25=Column[_col25], VALUE._col26=Column[_col26], VALUE._col27=Column[_col27], VALUE._col28=Column[_col28], VALUE._col29=Column[_col29], VALUE._col20=Column[_col20], VALUE._col21=Column[_col21], VALUE._col22=Column[_col22], VALUE._col23=Column[_col23], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]} 2020-06-30T09:46:08,812 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 24 newColExprMap: {VALUE._col30=Column[_col30], VALUE._col31=Column[_col31], VALUE._col10=Column[_col10], VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col24=Column[_col24], VALUE._col25=Column[_col25], VALUE._col26=Column[_col26], VALUE._col27=Column[_col27], VALUE._col28=Column[_col28], VALUE._col29=Column[_col29], VALUE._col20=Column[_col20], VALUE._col21=Column[_col21], VALUE._col22=Column[_col22], VALUE._col23=Column[_col23], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]} 2020-06-30T09:46:08,812 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 32 oldColExprMap: {VALUE._col30=Column[_col30], VALUE._col31=Column[_col31], VALUE._col10=Column[_col10], VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col24=Column[_col24], VALUE._col25=Column[_col25], VALUE._col26=Column[_col26], VALUE._col27=Column[_col27], VALUE._col28=Column[_col28], VALUE._col20=Column[_col20], VALUE._col21=Column[_col21], VALUE._col22=Column[_col22], VALUE._col23=Column[_col23], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 32 newColExprMap: {VALUE._col30=Column[_col30], VALUE._col31=Column[_col31], VALUE._col10=Column[_col10], VALUE._col11=Column[_col11], VALUE._col12=Column[_col12], VALUE._col24=Column[_col24], VALUE._col25=Column[_col25], VALUE._col26=Column[_col26], VALUE._col27=Column[_col27], VALUE._col28=Column[_col28], VALUE._col20=Column[_col20], VALUE._col21=Column[_col21], VALUE._col22=Column[_col22], VALUE._col23=Column[_col23], VALUE._col2=Column[_col2], VALUE._col3=Column[_col3], VALUE._col4=Column[_col4], VALUE._col5=Column[_col5], VALUE._col0=Column[_col0], VALUE._col1=Column[_col1], VALUE._col13=Column[_col13], VALUE._col14=Column[_col14], VALUE._col15=Column[_col15], VALUE._col16=Column[_col16], VALUE._col6=Column[_col6], VALUE._col17=Column[_col17], VALUE._col7=Column[_col7], VALUE._col18=Column[_col18], VALUE._col8=Column[_col8], VALUE._col19=Column[_col19], VALUE._col9=Column[_col9]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 39 oldColExprMap: {KEY._col0=Column[_col0], VALUE._col0=Column[_col1]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 39 newColExprMap: {KEY._col0=Column[_col0], VALUE._col0=Column[_col1]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 30 oldColExprMap: {VALUE._col30=Const string 2020-06-30 09:46:08, VALUE._col10=Column[_col11], VALUE._col11=Column[_col12], VALUE._col12=Column[_col13], VALUE._col24=Column[_col25], VALUE._col25=Column[_col26], VALUE._col26=Column[_col27], VALUE._col27=Column[_col28], VALUE._col28=Column[_col30], VALUE._col29=Column[_col31], VALUE._col20=Column[_col21], VALUE._col21=Column[_col22], VALUE._col22=Column[_col23], VALUE._col23=Column[_col24], KEY.reducesinkkey0=Column[_col0], VALUE._col2=Column[_col3], VALUE._col3=Column[_col4], VALUE._col4=Column[_col5], VALUE._col5=Column[_col6], VALUE._col0=Column[_col1], VALUE._col1=Column[_col2], VALUE._col13=Column[_col14], VALUE._col14=Column[_col15], VALUE._col15=Column[_col16], VALUE._col16=Column[_col17], VALUE._col6=Column[_col7], VALUE._col17=Column[_col18], VALUE._col7=Column[_col8], VALUE._col18=Column[_col19], VALUE._col8=Column[_col9], VALUE._col19=Column[_col20], VALUE._col9=Column[_col10]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 30 newColExprMap: {VALUE._col10=Column[_col11], VALUE._col11=Column[_col12], VALUE._col12=Column[_col13], VALUE._col24=Column[_col25], VALUE._col25=Column[_col26], VALUE._col26=Column[_col27], VALUE._col27=Column[_col28], VALUE._col28=Column[_col30], VALUE._col29=Column[_col31], VALUE._col20=Column[_col21], VALUE._col21=Column[_col22], VALUE._col22=Column[_col23], VALUE._col23=Column[_col24], KEY.reducesinkkey0=Column[_col0], VALUE._col2=Column[_col3], VALUE._col3=Column[_col4], VALUE._col4=Column[_col5], VALUE._col5=Column[_col6], VALUE._col0=Column[_col1], VALUE._col1=Column[_col2], VALUE._col13=Column[_col14], VALUE._col14=Column[_col15], VALUE._col15=Column[_col16], VALUE._col16=Column[_col17], VALUE._col6=Column[_col7], VALUE._col17=Column[_col18], VALUE._col7=Column[_col8], VALUE._col18=Column[_col19], VALUE._col8=Column[_col9], VALUE._col19=Column[_col20], VALUE._col9=Column[_col10]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 30 oldColExprMap: {VALUE._col10=Column[_col11], VALUE._col11=Column[_col12], VALUE._col12=Column[_col13], VALUE._col24=Column[_col25], VALUE._col25=Column[_col26], VALUE._col26=Column[_col27], VALUE._col27=Column[_col28], VALUE._col28=Column[_col30], VALUE._col29=Column[_col31], VALUE._col20=Column[_col21], VALUE._col21=Column[_col22], VALUE._col22=Column[_col23], VALUE._col23=Column[_col24], KEY.reducesinkkey0=Column[_col0], VALUE._col2=Column[_col3], VALUE._col3=Column[_col4], VALUE._col4=Column[_col5], VALUE._col5=Column[_col6], VALUE._col0=Column[_col1], VALUE._col1=Column[_col2], VALUE._col13=Column[_col14], VALUE._col14=Column[_col15], VALUE._col15=Column[_col16], VALUE._col16=Column[_col17], VALUE._col6=Column[_col7], VALUE._col17=Column[_col18], VALUE._col7=Column[_col8], VALUE._col18=Column[_col19], VALUE._col8=Column[_col9], VALUE._col19=Column[_col20], VALUE._col9=Column[_col10]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 30 newColExprMap: {VALUE._col10=Column[_col11], VALUE._col11=Column[_col12], VALUE._col12=Column[_col13], VALUE._col24=Column[_col25], VALUE._col25=Column[_col26], VALUE._col26=Column[_col27], VALUE._col27=Column[_col28], VALUE._col28=Column[_col30], VALUE._col29=Column[_col31], VALUE._col20=Column[_col21], VALUE._col21=Column[_col22], VALUE._col22=Column[_col23], VALUE._col23=Column[_col24], KEY.reducesinkkey0=Column[_col0], VALUE._col2=Column[_col3], VALUE._col3=Column[_col4], VALUE._col4=Column[_col5], VALUE._col5=Column[_col6], VALUE._col0=Column[_col1], VALUE._col1=Column[_col2], VALUE._col13=Column[_col14], VALUE._col14=Column[_col15], VALUE._col15=Column[_col16], VALUE._col16=Column[_col17], VALUE._col6=Column[_col7], VALUE._col17=Column[_col18], VALUE._col7=Column[_col8], VALUE._col18=Column[_col19], VALUE._col8=Column[_col9], VALUE._col19=Column[_col20], VALUE._col9=Column[_col10]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - JOIN 16 oldExprs: {0=[Column[KEY.reducesinkkey0], Column[VALUE._col0], Column[KEY.reducesinkkey2], Column[KEY.reducesinkkey3], Column[KEY.reducesinkkey4], Column[KEY.reducesinkkey5], Column[VALUE._col1], Column[VALUE._col2], Column[VALUE._col3], Column[VALUE._col4], Column[VALUE._col5], Column[VALUE._col6], Column[VALUE._col7], Column[VALUE._col8], Column[VALUE._col9], Column[VALUE._col10], Column[VALUE._col11], Column[VALUE._col12], Column[VALUE._col13], Column[VALUE._col14], Column[VALUE._col15], Column[VALUE._col16], Column[VALUE._col17], Column[VALUE._col18], Column[VALUE._col19], Column[VALUE._col20], Column[VALUE._col21], Column[VALUE._col22], Column[VALUE._col23], Column[VALUE._col24], Column[VALUE._col25], Column[VALUE._col26], Column[VALUE._col27], Column[VALUE._col28], Column[VALUE._col29]], 1=[Column[KEY.reducesinkkey0], Column[KEY.reducesinkkey1], Column[KEY.reducesinkkey2], Column[KEY.reducesinkkey3], Column[KEY.reducesinkkey4], Column[KEY.reducesinkkey5], Column[KEY.reducesinkkey6], Column[VALUE._col0], Column[VALUE._col1], Column[VALUE._col2], Column[VALUE._col3], Column[VALUE._col4], Column[VALUE._col5], Column[VALUE._col6], Column[VALUE._col7], Column[VALUE._col8], Column[VALUE._col9], Column[VALUE._col10], Column[VALUE._col11], Column[VALUE._col12], Column[VALUE._col13]]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - JOIN 16 newExprs: {0=[Column[KEY.reducesinkkey0], Column[VALUE._col0], Column[KEY.reducesinkkey2], Column[KEY.reducesinkkey3], Column[KEY.reducesinkkey4], Column[KEY.reducesinkkey5], Column[VALUE._col1], Column[VALUE._col2], Column[VALUE._col3], Column[VALUE._col5], Column[VALUE._col6], Column[VALUE._col13], Column[VALUE._col14], Column[VALUE._col15], Column[VALUE._col19], Column[VALUE._col20], Column[VALUE._col21], Column[VALUE._col29]], 1=[Column[KEY.reducesinkkey0], Column[KEY.reducesinkkey1], Column[KEY.reducesinkkey2], Column[KEY.reducesinkkey3], Column[KEY.reducesinkkey4], Column[KEY.reducesinkkey5], Column[KEY.reducesinkkey6], Column[VALUE._col0], Column[VALUE._col1], Column[VALUE._col2], Column[VALUE._col3], Column[VALUE._col4], Column[VALUE._col5], Column[VALUE._col6], Column[VALUE._col7], Column[VALUE._col8], Column[VALUE._col9], Column[VALUE._col10], Column[VALUE._col11], Column[VALUE._col12], Column[VALUE._col13]]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 14 oldColExprMap: {VALUE._col10=Column[unrestrictedstock], VALUE._col11=Column[qualitystock], VALUE._col12=Column[blockedstock], VALUE._col24=Column[odq_changemode], VALUE._col25=Column[timestmp], VALUE._col26=Column[tdate], VALUE._col27=Column[BLOCK__OFFSET__INSIDE__FILE], VALUE._col28=Column[INPUT__FILE__NAME], VALUE._col29=Column[ROW__ID], VALUE._col20=Column[previousstockintransfer], VALUE._col21=Column[previousrestrictedstock], VALUE._col22=Column[origin], VALUE._col23=Column[udp], KEY.reducesinkkey0=Column[plant], KEY.reducesinkkey1=Column[product], KEY.reducesinkkey2=Column[batch], KEY.reducesinkkey3=Column[storagelocation], KEY.reducesinkkey4=Column[specialstockindicator], KEY.reducesinkkey5=Column[salesdocument], KEY.reducesinkkey6=Column[salesdocumentitem], VALUE._col2=Column[customer], VALUE._col3=Column[vendor], VALUE._col4=Column[creationdate], VALUE._col5=Column[lastchangedate], VALUE._col0=Column[product], VALUE._col1=Column[salesdocumentitem], VALUE._col13=Column[returns], VALUE._col14=Column[stockintransfer], VALUE._col15=Column[restrictedstock], VALUE._col16=Column[previousunrestrictedstock], VALUE._col6=Column[deletionflag], VALUE._col17=Column[previousqualitystock], VALUE._col7=Column[blockedinventoryindicator], VALUE._col18=Column[previousblockedstock], VALUE._col8=Column[currentyear], VALUE._col19=Column[previousreturns], VALUE._col9=Column[currentperiod]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 14 newColExprMap: {VALUE._col29=Column[ROW__ID], VALUE._col20=Column[previousstockintransfer], VALUE._col21=Column[previousrestrictedstock], KEY.reducesinkkey0=Column[plant], KEY.reducesinkkey1=Column[product], KEY.reducesinkkey2=Column[batch], KEY.reducesinkkey3=Column[storagelocation], KEY.reducesinkkey4=Column[specialstockindicator], KEY.reducesinkkey5=Column[salesdocument], KEY.reducesinkkey6=Column[salesdocumentitem], VALUE._col2=Column[customer], VALUE._col3=Column[vendor], VALUE._col5=Column[lastchangedate], VALUE._col0=Column[product], VALUE._col1=Column[salesdocumentitem], VALUE._col13=Column[returns], VALUE._col14=Column[stockintransfer], VALUE._col15=Column[restrictedstock], VALUE._col6=Column[deletionflag], VALUE._col19=Column[previousreturns]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 15 oldColExprMap: {VALUE._col10=Column[_col17], VALUE._col11=Column[_col18], VALUE._col12=Column[_col19], KEY.reducesinkkey0=Column[_col0], KEY.reducesinkkey1=Column[_col1], KEY.reducesinkkey2=Column[_col2], KEY.reducesinkkey3=Column[_col3], KEY.reducesinkkey4=Column[_col4], KEY.reducesinkkey5=Column[_col5], KEY.reducesinkkey6=Column[_col6], VALUE._col2=Column[_col9], VALUE._col3=Column[_col10], VALUE._col4=Column[_col11], VALUE._col5=Column[_col12], VALUE._col0=Column[_col7], VALUE._col1=Column[_col8], VALUE._col13=Column[_col20], VALUE._col6=Column[_col13], VALUE._col7=Column[_col14], VALUE._col8=Column[_col15], VALUE._col9=Column[_col16]} 2020-06-30T09:46:08,813 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 15 newColExprMap: {VALUE._col10=Column[_col17], VALUE._col11=Column[_col18], VALUE._col12=Column[_col19], KEY.reducesinkkey0=Column[_col0], KEY.reducesinkkey1=Column[_col1], KEY.reducesinkkey2=Column[_col2], KEY.reducesinkkey3=Column[_col3], KEY.reducesinkkey4=Column[_col4], KEY.reducesinkkey5=Column[_col5], KEY.reducesinkkey6=Column[_col6], VALUE._col2=Column[_col9], VALUE._col3=Column[_col10], VALUE._col4=Column[_col11], VALUE._col5=Column[_col12], VALUE._col0=Column[_col7], VALUE._col1=Column[_col8], VALUE._col13=Column[_col20], VALUE._col6=Column[_col13], VALUE._col7=Column[_col14], VALUE._col8=Column[_col15], VALUE._col9=Column[_col16]} 2020-06-30T09:46:08,814 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 6 oldColExprMap: {VALUE._col10=Column[_col17], VALUE._col11=Column[_col18], VALUE._col12=Column[_col19], KEY.reducesinkkey0=Column[_col2], KEY.reducesinkkey1=Column[_col3], KEY.reducesinkkey2=Column[_col1], KEY.reducesinkkey3=Column[_col6], KEY.reducesinkkey4=Column[_col4], KEY.reducesinkkey5=Column[_col5], KEY.reducesinkkey6=Column[_col0], KEY.reducesinkkey7=Column[_col20], VALUE._col2=Column[_col9], KEY.reducesinkkey8=Column[_col21], VALUE._col3=Column[_col10], VALUE._col4=Column[_col11], VALUE._col5=Column[_col12], VALUE._col0=Column[_col7], VALUE._col1=Column[_col8], VALUE._col6=Column[_col13], VALUE._col7=Column[_col14], VALUE._col8=Column[_col15], VALUE._col9=Column[_col16]} 2020-06-30T09:46:08,814 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 6 newColExprMap: {VALUE._col10=Column[_col17], VALUE._col11=Column[_col18], VALUE._col12=Column[_col19], KEY.reducesinkkey0=Column[_col2], KEY.reducesinkkey1=Column[_col3], KEY.reducesinkkey2=Column[_col1], KEY.reducesinkkey3=Column[_col6], KEY.reducesinkkey4=Column[_col4], KEY.reducesinkkey5=Column[_col5], KEY.reducesinkkey6=Column[_col0], KEY.reducesinkkey7=Column[_col20], VALUE._col2=Column[_col9], KEY.reducesinkkey8=Column[_col21], VALUE._col3=Column[_col10], VALUE._col4=Column[_col11], VALUE._col5=Column[_col12], VALUE._col0=Column[_col7], VALUE._col1=Column[_col8], VALUE._col6=Column[_col13], VALUE._col7=Column[_col14], VALUE._col8=Column[_col15], VALUE._col9=Column[_col16]} 2020-06-30T09:46:08,814 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 1 oldColExprMap: {VALUE._col10=Column[lfmon], VALUE._col11=Column[kalab], VALUE._col12=Column[kains], VALUE._col20=Column[date_migration], VALUE._col21=Column[BLOCK__OFFSET__INSIDE__FILE], VALUE._col22=Column[INPUT__FILE__NAME], VALUE._col23=Column[ROW__ID], KEY.reducesinkkey0=Const int 0, VALUE._col2=Column[charg], VALUE._col3=Column[lgort], VALUE._col4=Column[sobkz], VALUE._col5=Column[vbeln], VALUE._col0=Column[werks], VALUE._col1=Column[matnr], VALUE._col13=Column[kaspe], VALUE._col14=Column[kavla], VALUE._col15=Column[kavin], VALUE._col16=Column[kavsp], VALUE._col6=Column[posnr], VALUE._col17=Column[origin], VALUE._col7=Column[ersda], VALUE._col18=Column[udp], VALUE._col8=Column[kaspr], VALUE._col19=Column[odq_changemode], VALUE._col9=Column[lfgja]} 2020-06-30T09:46:08,814 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ColumnPrunerProcFactory (:()) - RS 1 newColExprMap: {VALUE._col10=Column[lfmon], VALUE._col11=Column[kalab], VALUE._col12=Column[kains], VALUE._col20=Column[date_migration], KEY.reducesinkkey0=Const int 0, VALUE._col2=Column[charg], VALUE._col3=Column[lgort], VALUE._col4=Column[sobkz], VALUE._col5=Column[vbeln], VALUE._col0=Column[werks], VALUE._col1=Column[matnr], VALUE._col13=Column[kaspe], VALUE._col14=Column[kavla], VALUE._col15=Column[kavin], VALUE._col16=Column[kavsp], VALUE._col6=Column[posnr], VALUE._col17=Column[origin], VALUE._col7=Column[ersda], VALUE._col18=Column[udp], VALUE._col8=Column[kaspr], VALUE._col19=Column[odq_changemode], VALUE._col9=Column[lfgja]} 2020-06-30T09:46:08,814 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: correlation.AbstractCorrelationProcCtx (:()) - Overriding hive.optimize.reducededuplication.min.reducer to 1 due to a write to transactional table(s) refining_silver_test.factinventory,refining_silver_test.factinventory 2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.SetReducerParallelism (:()) - Set parallelism for reduce sink RS[1] to: 16 2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.SetReducerParallelism (:()) - Set parallelism for reduce sink RS[14] to: 84 2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.SetReducerParallelism (:()) - Set parallelism for reduce sink RS[6] to: 16 2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.SetReducerParallelism (:()) - Set parallelism for reduce sink RS[15] to: 84 2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Memory monitor info set to : { isLlap: false executorsPerNode: 1 maxExecutorsOverSubscribeMemory: 3 memoryOverSubscriptionFactor: 0.20000000298023224 memoryCheckInterval: 100000 noConditionalTaskSize: 1717986918 adjustedNoConditionalTaskSize: 1717986918 hashTableInflationFactor: 2.0 threshold: 3435973836 } 2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - maxJoinMemory: 1717986918 2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Estimated number of buckets 1 2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Join input#0; onlineDataSize: 1870875047; Statistics: Num rows: 3287943 Data size: 5097628704 Basic stats: COMPLETE Column stats: NONE 2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Try dynamic partitioned hash join with estimated 84 reducers 2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Join input#0; onlineDataSize: 1870875047; Statistics: Num rows: 3287943 Data size: 5097628704 Basic stats: COMPLETE Column stats: NONE 2020-06-30T09:46:08,840 INFO [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: optimizer.ConvertJoinMapJoin (:()) - Join input#1; onlineDataSize: 388737795; Statistics: Num rows: 133039 Data size: 522108828 Basic stats: COMPLETE Column stats: NONE 2020-06-30T09:46:08,841 ERROR [16ba3bf1-1774-4bcd-b0a0-1d122df23311 HiveServer2-Handler-Pool: Thread-177864]: ql.Driver (:()) - FAILED: NullPointerException null java.lang.NullPointerException at org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc.<init>(ExprNodeColumnDesc.java:84) at org.apache.hadoop.hive.ql.plan.ExprNodeColumnDesc.<init>(ExprNodeColumnDesc.java:80) at org.apache.hadoop.hive.ql.plan.ExprNodeDescUtils.resolveJoinKeysAsRSColumns(ExprNodeDescUtils.java:450) at org.apache.hadoop.hive.ql.plan.ExprNodeDescUtils.resolveJoinKeysAsRSColumns(ExprNodeDescUtils.java:411) at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.getMapJoinDesc(MapJoinProcessor.java:1103) at org.apache.hadoop.hive.ql.optimizer.MapJoinProcessor.convertJoinOpMapJoinOp(MapJoinProcessor.java:372) at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.convertJoinMapJoin(ConvertJoinMapJoin.java:1194) at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.convertJoinDynamicPartitionedHashJoin(ConvertJoinMapJoin.java:1417) at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.fallbackToReduceSideJoin(ConvertJoinMapJoin.java:1449) at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.checkAndConvertSMBJoin(ConvertJoinMapJoin.java:491) at org.apache.hadoop.hive.ql.optimizer.ConvertJoinMapJoin.process(ConvertJoinMapJoin.java:158) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89) at org.apache.hadoop.hive.ql.lib.ForwardWalker.walk(ForwardWalker.java:74) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120) at org.apache.hadoop.hive.ql.parse.TezCompiler.runStatsDependentOptimizations(TezCompiler.java:434) at org.apache.hadoop.hive.ql.parse.TezCompiler.optimizeOperatorPlan(TezCompiler.java:170) at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:148) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12521) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11855) at org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:66) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) at org.apache.hadoop.hive.ql.parse.MergeSemanticAnalyzer.analyzeMerge(MergeSemanticAnalyzer.java:300) at org.apache.hadoop.hive.ql.parse.MergeSemanticAnalyzer.analyze(MergeSemanticAnalyzer.java:62) at org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer.analyzeInternal(RewriteSemanticAnalyzer.java:71) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:289) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1870) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1817) at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1812) at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126) at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:197) at org.apache.hive.service.cli.operation.SQLOperation.runInternal(SQLOperation.java:262) at org.apache.hive.service.cli.operation.Operation.run(Operation.java:260) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementInternal(HiveSessionImpl.java:575) at org.apache.hive.service.cli.session.HiveSessionImpl.executeStatementAsync(HiveSessionImpl.java:561) at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) at com.sun.proxy.$Proxy102.executeStatementAsync(Unknown Source) at org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:315) at org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:566) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1557) at org.apache.hive.service.rpc.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1542) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) at org.apache.hive.service.auth.TSetIpAddressProcessor.process(TSetIpAddressProcessor.java:56) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
... View more
07-01-2020
02:31 AM
Hello,
For serveral days I encountered an NPE error on hive when I execute several merge on the same table. When I execute the first merge on the table its ok but when i execute the second merge i get this error "Error: Error while compiling statement: FAILED: NullPointerException null (state=42000,code=40000)".
After many searches, i have try this in my code:
set hive.auto.convert.join=false;
and this resolved my NPE error but I don't understand why hive.auto.convert.join = true makes this error, knowing that this parameter checks if the smaller table file size is greater than the value specified by hive.mapjoin .smalltable.filesize.
Can someone help me to understand this strange behaviour ?
Thanks in advance
Best Regard
... View more
Labels:
- Labels:
-
Apache Hive
04-14-2020
07:54 AM
1 Kudo
Hello, @Eric_B , thanks for the reply, after few search me to I have no permanente solution but for overcome this issue we work with insert (for all data) and make update gestion with materialized view hive. And work fine and fast for my usecase. Best Regards
... View more
04-08-2020
06:13 AM
1 Kudo
Hi, after some researches i have find a solution to this issues. The problem was from the Hive table definition for storing data. I was defining some properties of my table like this : hive.createTable("res_idos_0")
.ifNotExists()
.prop("serialization.encoding","UTF-8")
.prop("escape.delim" , "\t")
.column("t_date","TIMESTAMP") But when we are in writeStream and we use special characters, the use of property escape.delim is note supported and we can't save characters correctly. So, i have removed the property escape.delim in my hive table definition and i had also added this line in my code for being certain that file save in HDFS have the right encoding. System.setProperty("file.encoding", "UTF-8")
... View more
04-06-2020
04:17 AM
Hello,
I'm facing an issue with the display and storage of special charactere in hive.
I'm using spark for doing a WriteStream like this in Hive,
// Write result in hive
val query = trimmedDF.writeStream
//.format("console")
.format("com.hortonworks.spark.sql.hive.llap.streaming.HiveStreamingDataSource")
.outputMode("append")
.option("metastoreUri", metastoreUri)
.option("database", "dwh_prod")
.option("table", "res_idos_0")
.option("checkpointLocation", "/tmp/idos_LVD_060420_0")
.queryName("test_final")
.option("truncate", "false")
.option("encoding", "UTF-8")
.start()
query.awaitTermination()
but when I have a special charactere Hive doesn't display it. I have already fixe encoding UTF8 in the hive table :
select distinct(analyte) from res_idos_0;
+--------------------------------------------+
| analyte |
+--------------------------------------------+
| D02 |
| E |
| E - Hauteur Int��rieure jupe - 6,75mm |
| Hauteur totale |
| Long tube apparent (embout 408 assembl��) |
| Side streaming - poids apr��s |
| Tenue tube plongeur |
| 1 dose - poids avant |
| Diam��tre 1er joint de sertissage |
| HDS - Saillie Point Mort Bas |
| P - Epaisseur tourette P5 - 0,51mm |
+--------------------------------------------+
But if I display the data in console with writeStream the special chararacter are correctly display or if I use write fonction for write in hive like this:
final_DF.write.format("com.hortonworks.spark.sql.hive.llap.HiveWarehouseConnector")
.mode("overwrite")
.option("table","dwh_prod.result_idos_lims3")
.save()
The charactere are correctly display in hive
+-------------------------------------------+
| analyte |
+-------------------------------------------+
| 1 dose |
| 1 dose (moyenne) - Kinf |
| 1 dose (écart type) |
| 1 dose - poids avant |
| 1 dose individuelle (maxi) |
| 1,00mm |
| 1,3,5-trioxane |
I use spark 2.3.2 an hive 3.1.0
Those anyone face this issue or have clue or a solution for me.
Thanks in advance,
Best Regards
... View more
Labels:
03-06-2020
08:28 AM
1 Kudo
Hello @pal_1990, I think your input is something like this: +----------------------------------------------------+
| semicolon.a |
+----------------------------------------------------+
| 1;13004211,13004211_02_13004212,4000000003378605589,1105,2000 |
+----------------------------------------------------+ 1 . You need to separate the one from other values, for this I use posexplode fonction: select pe.i,pe.x from semicolon lateral view posexplode(split(a,';')) pe as i,x;
+-------+----------------------------------------------------+
| pe.i | pe.x |
+-------+----------------------------------------------------+
| 0 | 1 |
| 1 | 13004211,13004211_02_13004212,4000000003378605589,1105,2000 |
+-------+----------------------------------------------------+ 2. You on only select where pe.i =1: select t.x from
(select pe.i,pe.x
from semicolon lateral view posexplode(split(a,';')) pe as i,x) t where t.i=1 ; +----------------------------------------------------+ | t.x | +----------------------------------------------------+ | 13004211,13004211_02_13004212,4000000003378605589,1105,2000 | +----------------------------------------------------+
3. You split values in columns; select split(t.x,',')[0] as col1,
split(t.x,',')[1] as col2,
split(t.x,',')[2] as col3,
split(t.x,',')[3] as col4,
split(t.x,',')[4] as col5
from
(select pe.i,pe.x
from semicolon lateral view posexplode(split(a,';')) pe as i,x) t where t.i=1 ; +-----------+-----------------------+----------------------+-------+-------+ | col1 | col2 | col3 | col4 | col5 | +-----------+-----------------------+----------------------+-------+-------+ | 13004211 | 13004211_02_13004212 | 4000000003378605589 | 1105 | 2000 | +-----------+-----------------------+----------------------+-------+-------+ I hope it will help you. Best regards
... View more
02-17-2020
08:21 AM
Hi, @rajkumar_singh thanks to your advice I found my file hiveserver2 Interactive.log and the error is in task 4-5. I noticed 2 things : - First, unfortunately before the first NPE error in the file hiveserver2 Interactive.log it is another NPE that we find - Secondly, in task 3 we have - Cannot get a table snapshot for factmachine_mv but I don't thinks that is the error's source. HiveServeur2Interactive.log: 2020-02-17T15:11:13,525 INFO [HiveServer2-Background-Pool: Thread-5750]: FileOperations (FSStatsAggregator.java:aggregateStats(101)) - Read stats for : dwh_dev.factmachine_mv/ numRows 1231370
2020-02-17T15:11:13,525 INFO [HiveServer2-Background-Pool: Thread-5750]: FileOperations (FSStatsAggregator.java:aggregateStats(101)) - Read stats for : dwh_dev.factmachine_mv/ rawDataSize 552885130
2020-02-17T15:11:13,525 WARN [HiveServer2-Background-Pool: Thread-5750]: metadata.Hive (Hive.java:alterTable(778)) - Cannot get a table snapshot for factmachine_mv
2020-02-17T15:11:13,609 INFO [HiveServer2-Background-Pool: Thread-5750]: stats.BasicStatsTask (SessionState.java:printInfo(1227)) - Table dwh_dev.factmachine_mv stats: [numFiles=2, numRows=1231370, totalSize=1483151, rawDataSize=552885130]
2020-02-17T15:11:13,609 INFO [HiveServer2-Background-Pool: Thread-5750]: stats.BasicStatsTask (BasicStatsTask.java:aggregateStats(271)) - Table dwh_dev.factmachine_mv stats: [numFiles=2, numRows=1231370, totalSize=1483151, rawDataSize=552885130]
2020-02-17T15:11:13,620 INFO [HiveServer2-Background-Pool: Thread-5750]: mapred.FileInputFormat (FileInputFormat.java:listStatus(259)) - Total input files to process : 1
2020-02-17T15:11:14,881 INFO [HiveServer2-Background-Pool: Thread-5750]: ql.Driver (Driver.java:launchTask(2710)) - Starting task [Stage-5:DDL] in serial mode
2020-02-17T15:11:14,902 INFO [HiveServer2-Background-Pool: Thread-5750]: parse.CalcitePlanner (CalcitePlanner.java:genLogicalPlan(385)) - Starting generating logical plan
2020-02-17T15:11:14,904 INFO [HiveServer2-Background-Pool: Thread-5750]: parse.CalcitePlanner (SemanticAnalyzer.java:genResolvedParseTree(12232)) - Completed phase 1 of Semantic Analysis
2020-02-17T15:11:14,904 INFO [HiveServer2-Background-Pool: Thread-5750]: parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(2113)) - Get metadata for source tables
2020-02-17T15:11:14,904 INFO [HiveServer2-Background-Pool: Thread-5750]: parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(2244)) - Get metadata for subqueries
2020-02-17T15:11:14,905 INFO [HiveServer2-Background-Pool: Thread-5750]: parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(2113)) - Get metadata for source tables
2020-02-17T15:11:14,935 INFO [HiveServer2-Background-Pool: Thread-5750]: parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(2244)) - Get metadata for subqueries
2020-02-17T15:11:14,936 INFO [HiveServer2-Background-Pool: Thread-5750]: parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(2268)) - Get metadata for destination tables
2020-02-17T15:11:14,936 INFO [HiveServer2-Background-Pool: Thread-5750]: parse.CalcitePlanner (SemanticAnalyzer.java:getMetaData(2268)) - Get metadata for destination tables
2020-02-17T15:11:14,944 INFO [HiveServer2-Background-Pool: Thread-5750]: ql.Context (Context.java:getMRScratchDir(551)) - New scratch dir is hdfs://lvdcluster/tmp/hive/hive/c0988fe9-1a5e-4a6a-9ddd-7b782d0cdba1/hive_2020-02-17_15-11-14_897_3869104234201454658-21
2020-02-17T15:11:14,945 INFO [HiveServer2-Background-Pool: Thread-5750]: parse.CalcitePlanner (SemanticAnalyzer.java:genResolvedParseTree(12237)) - Completed getting MetaData in Semantic Analysis
2020-02-17T15:11:14,945 INFO [HiveServer2-Background-Pool: Thread-5750]: parse.BaseSemanticAnalyzer (CalcitePlanner.java:canCBOHandleAst(875)) - Not invoking CBO because the statement has lateral views
2020-02-17T15:11:14,946 ERROR [HiveServer2-Background-Pool: Thread-5750]: exec.TaskRunner (TaskRunner.java:runSequential(108)) - Error in executeTask
java.lang.NullPointerException: null
at org.apache.calcite.plan.RelOptMaterialization.<init>(RelOptMaterialization.java:68) ~[calcite-core-1.16.0.3.1.4.0-315.jar:1.16.0.3.1.4.0-315]
at org.apache.hadoop.hive.ql.metadata.HiveMaterializedViewsRegistry.addMaterializedView(HiveMaterializedViewsRegistry.java:235) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hadoop.hive.ql.metadata.HiveMaterializedViewsRegistry.createMaterializedView(HiveMaterializedViewsRegistry.java:187) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hadoop.hive.ql.exec.MaterializedViewTask.execute(MaterializedViewTask.java:59) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212) ~[hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:103) [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2712) [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2383) [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2055) [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1753) [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1747) [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157) [hive-exec-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226) [hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87) [hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:324) [hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at java.security.AccessController.doPrivileged(Native Method) [?:1.8.0_112]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_112]
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730) [hadoop-common-3.1.1.3.1.4.0-315.jar:?]
at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:342) [hive-service-3.1.0.3.1.4.0-315.jar:3.1.0.3.1.4.0-315]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_112]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_112]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_112]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_112]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
2020-02-17T15:11:14,947 INFO [HiveServer2-Background-Pool: Thread-5750]: reexec.ReOptimizePlugin (ReOptimizePlugin.java:run(70)) - ReOptimization: retryPossible: false
2020-02-17T15:11:14,948 ERROR [HiveServer2-Background-Pool: Thread-5750]: ql.Driver (SessionState.java:printError(1250)) - FAILED: Hive Internal Error: org.apache.hadoop.hive.ql.metadata.HiveException(Error while invoking FailureHook. hooks: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.reexec.ReExecutionOverlayPlugin$LocalHook.run(ReExecutionOverlayPlugin.java:45)
at org.apache.hadoop.hive.ql.HookRunner.invokeGeneralHook(HookRunner.java:296)
at org.apache.hadoop.hive.ql.HookRunner.runFailureHooks(HookRunner.java:283)
at org.apache.hadoop.hive.ql.Driver.invokeFailureHooks(Driver.java:2664)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:2434)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:2055)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1753)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1747)
at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:157)
at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:226)
at org.apache.hive.service.cli.operation.SQLOperation.access$700(SQLOperation.java:87)
at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:324)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:342)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
) For the moment, my solution is to use use a simple view. Because I noticed that even if the materilized view doesn't work, a simple view work with this query.
... View more