Hadoop sequence file EOFException

idobrt — Fri, 16 Sep 2022 10:45:08 GMT

Hello. I tryed to process image on hadoop and i using HVPI (https://github.com/xmpy/hvpi). It's open source. I using it to extract the frames from video but it not provide a output format to save the frames on HDFS. So i tryed to use the SequencefileOutputFormat to save the frames on HDFS. I readed a book and make some changes to the code work on hadoop 2.7.1. Apparently it's worked but when i tryed to recover the file in another mapreduce job i got EOFException. I thinked there is 3 possibily:

1- My SequenceFileOutputFormat is wrong and save a corrupted file;

2-My SequenceInputFormt is wrong;

3- The HVPI custom type is wrong.

The code to save a Sequence file:

public static void main(String[] args) throws Exception {

//判断输入参数

if (args.length != 2) {

System.err.println("Usage: <input path> <output path>");

System.exit(-1);

}

Configuration conf = new Configuration();

//设置Master URL

conf.set("fs.defaultFS","hdfs://evoido:9000");

Job job = new Job(conf, "MRVideoReader");

job.setJarByClass(MRVideoReader.class);

Path in = new Path(args[0]);

Path out = new Path(args[1]);

//reduce task zero job is only map

job.setJobName("Sequence Writer Test");

job.setNumReduceTasks(0);

job.setInputFormatClass(VideoInputFormat.class);

job.setMapperClass(MRVideoReaderMapper.class);

job.setMapOutputKeyClass(Text.class);

//job.setMapOutputValueClass(Text.class);

job.setMapOutputValueClass(ImageWritable.class);

job.setReducerClass(MRVideoReaderReducer.class);

job.setOutputFormatClass(SequenceFileOutputFormat.class);

job.setJarByClass(MRVideoReader.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(ImageWritable.class);

//job.setOutputValueClass(Text.class);

FileInputFormat.addInputPath(job,in);

SequenceFileOutputFormat.setOutputPath(job, out);

System.exit(job.waitForCompletion(true)?0:1);

}

The code to read from HDFS:

public static void main(String[] args) throws Exception {

//判断输入参数

if (args.length != 2) {

System.err.println("Usage: <input path> <output path>");

System.exit(-1);

}

Configuration conf = new Configuration();

//设置Master URL

conf.set("fs.defaultFS","hdfs://evoido:9000");

Job job = new Job(conf, "MRVideoReader");

job.setJarByClass(MRVideoReader.class);

Path in = new Path(args[0]);

Path out = new Path(args[1]);

//reduce task zero job is only map

job.setJobName("Sequence Writer Test");

job.setNumReduceTasks(0);

job.setInputFormatClass(SequenceFileInputFormat.class);

job.setMapperClass(MRVideoReaderMapper.class);

job.setMapOutputKeyClass(Text.class);

//job.setMapOutputValueClass(Text.class);

job.setMapOutputValueClass(ImageWritable.class);

job.setReducerClass(MRVideoReaderReducer.class);

job.setOutputFormatClass(TextOutputFormat.class);

job.setJarByClass(MRVideoReader.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(ImageWritable.class);

//job.setOutputValueClass(Text.class);

SequenceFileInputFormat.addInputPath(job, in);

FileOutputFormat.setOutputPath(job, out);

System.exit(job.waitForCompletion(true)?0:1);

}

This is the code's piece from the custom type.

Maybe the readFilds or write methods are incorrect.

public void readFields(DataInput in) throws IOException {

int len = WritableUtils.readVInt(in);

System.out.println("Valor de len:"+len);

byte[] temp = new byte[len];

in.readFully(temp, 0, len);

ByteArrayInputStream byteStream = new ByteArrayInputStream(temp); // 输入流；

bufferedImage = ImageIO.read(byteStream); // 从输入流中，读取图片存入image中，而这里in可以为ByteArrayInputStream();

}

public void write(DataOutput out) throws IOException {

ByteArrayOutputStream byteOutputStream = new ByteArrayOutputStream();

ImageIO.write( bufferedImage, "png", byteOutputStream );

WritableUtils.writeVInt(out, byteOutputStream.size());

}

Re: Hadoop sequence file EOFException

idobrt — Sun, 23 Oct 2016 03:40:22 GMT

I founded the error. The HVPI custom type was wrong. It was in the write method.

Re: Hadoop sequence file EOFException

idobrt — Sun, 23 Oct 2016 04:00:17 GMT

I founded the error. The HVPI custom type was wrong. I just put out.write(byteOutputStream.toByteArray()); on write method and now it's work.

question Re: Hadoop sequence file EOFException in Archives of Support Questions (Read Only)

Hadoop sequence file EOFException

Re: Hadoop sequence file EOFException

Re: Hadoop sequence file EOFException