my hive query is failing with NPE when I am querying column with timestamp on hive partitioned orc table.

[#1]

Caused by: java.lang.NullPointerException

at org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$TimestampStatisticsImpl.getMinimum(ColumnStatisticsImpl.java:795)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.getMin(RecordReaderImpl.java:2343)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.evaluatePredicate(RecordReaderImpl.java:2366)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.pickRowGroups(RecordReaderImpl.java:2564)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:2627)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:3060)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:3102)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:288)

at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:534)

at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.<init>(OrcRawRecordMerger.java:183)

[#2 · Accepted solution]

it seems that you are hitting https://issues.apache.org/jira/browse/HIVE-12349

can you try to run your query after setting hive.optimize.index.filter=false

[#3]

@a kumar

This can be relate to Hive Jira 12349: https://issues.apache.org/jira/browse/HIVE-12349

Can you please share the query being run?

Thanks and Regards,

Sindhu

[#4]

Hi I too face this issue(with tez and mr engines) now for below italized query - query to fetch a set of records from a view and insert into a table.

HDP Version : (2.3.2.0-2950) version in both dev and prod.

Hive Version : Hive 1.2.1.2.3.2.48-5

Same query worked for a week in production and 1st time i face this issue. is it due to incorrect data in Date Column?

insert overwrite table crhs_fmtrade_transient_break partition(transient_break_reporting_date, source_system)

select break_id, trade_id, reporting_date, break_type, first_seen_date, processed_date,transientbreakcloseddate, closuretype,transientbreakindicator, transientbreaktype, breakfinalsysteminvststatus, transientbreakprocesseddate, commentarysysteminvestigationstatus, transientbreakreportingdate, source_system from v_fmtrade_mxg_transient_break;

Error StackTrace (Hive on Tez):

Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$DateStatisticsImpl.getMinimum(ColumnStatisticsImpl.java:745) at org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$DateStatisticsImpl.getMinimum(ColumnStatisticsImpl.java:665) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.getMin(RecordReaderImpl.java:304) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.evaluatePredicateProto(RecordReaderImpl.java:332) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:710) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.pickRowGroups(RecordReaderImpl.java:751) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:777) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:986) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1019) at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:205) at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:598) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$ReaderPair.<init>(OrcRawRecordMerger.java:183) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger$OriginalReaderPair.<init>(OrcRawRecordMerger.java:226) at org.apache.hadoop.hive.ql.io.orc.OrcRawRecordMerger.<init>(OrcRawRecordMerger.java:437) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getReader(OrcInputFormat.java:1235) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1117) at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:249) ... 26 more

Error StackTrace (Hive on MR engine):

Caused by: java.lang.NullPointerException

at org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$DateStatisticsImpl.getMinimum(ColumnStatisticsImpl.java:745)

at org.apache.hadoop.hive.ql.io.orc.ColumnStatisticsImpl$DateStatisticsImpl.getMinimum(ColumnStatisticsImpl.java:665)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.getMin(RecordReaderImpl.java:304)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.evaluatePredicateProto(RecordReaderImpl.java:332)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:710)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.pickRowGroups(RecordReaderImpl.java:751)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:777)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:986)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceToNextRow(RecordReaderImpl.java:1019)

at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.<init>(RecordReaderImpl.java:205)

at org.apache.hadoop.hive.ql.io.orc.ReaderImpl.rowsOptions(ReaderImpl.java:598)

at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.createReaderFromFile(OrcInputFormat.java:230)

at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat$OrcRecordReader.<init>(OrcInputFormat.java:163)

at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getRecordReader(OrcInputFormat.java:1108)

at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.<init>(CombineHiveRecordReader.java:67)

... 16 more