Hive error: java.io.EOFException: Unexpected end of input stream

[#1]

I am running an Hive query (my execution engine is Tez) on an external table with gz files. The error message i get is "java.io.EOFException: Unexpected end of input stream". I have done a check on HDFS for corrupt files and it returns a healthy status. Anyone come across a similar issue?

Caused by: java.io.EOFException: Unexpected end of input streamat org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:145)at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:85)at java.io.InputStream.read(InputStream.java:101)at org.apache.hadoop.util.LineReader.fillBuffer(LineReader.java:180)at org.apache.hadoop.util.LineReader.readDefaultLine(LineReader.java:216)at org.apache.hadoop.util.LineReader.readLine(LineReader.java:174)at org.apache.hadoop.mapred.LineRecordReader.skipUtfByteOrderMark(LineRecordReader.java:208)at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:246)at org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:48)at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)... 22 more

Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:90, Vertex vertex_1483105407610_10086_1_00

[#2 · Accepted solution]

The issue was that some files with zero file size were causing the error. The zero-sized files were moved out and issue was resolved.

[#3]

Check for more details and got error as : "Unexpected end of input stream"

Now, Get the hdfs LOCATION for the table by using below command on HUE or HIVE shell:

show create table <table-name>;

Check for the zero byte size files and remove them from hdfs location using below command:

hdfs dfs -rm -skipTrash $(hdfs dfs -ls -R <hdfs_location> | grep -v "^d" | awk '{if ($5 == 0) print $8}')

Try running again our query which ran successfully this time.