How to import table using Sqoop which has Blob column into Hive as Binary column in ORC File Format

Revathi91 — Fri, 16 Sep 2022 11:03:04 GMT

I had used the following sqoop command to import db2 data which is defined as BLOB into Hive in ORC format.

sqoop import \--connect "jdbc:db2://******/DB22" --username ***** --password '******' \--table temp --map-column-hive row_id=String,binary_id=Binary 
--map-column-java row_id=String  -m 1 
--hcatalog-database temp_database --hcatalog-table temp_table --create-hcatalog-table --hcatalog-storage-stanza "stored as orcfile"

1) Table is created in Hive (one column is having binary data,rest of the columns are in STRING format) and when I do ,

select count(*) from temp_table; --I am able to get the count

When I select without that one Binary column also ,I am able to pull the result.

But,
2) When I try to select something (including Binary column) from that table like below ,

select * from temp_table limit 1;

I am getting below error:

Error: java.io.IOException: java.lang.reflect.InvocationTargetException**        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderCreationException(HiveIOExceptionHandlerChain.java:97)        at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderCreationException(HiveIOExceptionHandlerUtil.java:57)        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.initNextRecordReader(HadoopShimsSecure.java:266)        at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.<init>(HadoopShimsSecure.java:213)

Caused by: java.lang.NegativeArraySizeException**        at org.apache.hadoop.hive.ql.io.orc.RecordReaderUtils.readDiskRanges(RecordReaderUtils.java:271)        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readPartialDataStreams(RecordReaderImpl.java:991)        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.readStripe(RecordReaderImpl.java:819)        at org.apache.hadoop.hive.ql.io.orc.RecordReaderImpl.advanceStripe(RecordReaderImpl.java:1013)

Could someone help me to resolve this issue?

Re: How to import table using Sqoop which has Blob column into Hive as Binary column in ORC File For

csguna — Sun, 12 Feb 2017 02:54:00 GMT

The application is trying to invoke an array with a negative size. for example -1 -2 etc

Could you please let me know your Hive ORC related configuration.

hive.exec.orc.skip.corrupt.data

hive.exec.orc.default.row.index.stride

Re: How to import table using Sqoop which has Blob column into Hive as Binary column in ORC File For

Revathi91 — Mon, 13 Feb 2017 00:22:59 GMT

Thanks for your reply.

Please find Hive ORC related config below.

hive.exec.orc.skip.corrupt.data=false
hive.exec.orc.default.row.index.stride=10000

question Re: How to import table using Sqoop which has Blob column into Hive as Binary column in ORC File For in Support Questions

How to import table using Sqoop which has Blob column into Hive as Binary column in ORC File Format

Re: How to import table using Sqoop which has Blob column into Hive as Binary column in ORC File For

Re: How to import table using Sqoop which has Blob column into Hive as Binary column in ORC File For