Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Hadoop sequence file EOFException

avatar
Explorer

 

Hello. I tryed to process image on hadoop and i using HVPI (https://github.com/xmpy/hvpi). It's open source. I using it to extract the frames from video but it not provide a output format to save the frames on HDFS. So i tryed to use the SequencefileOutputFormat to save the frames on HDFS. I readed a book and make some changes to the code work on hadoop 2.7.1. Apparently it's worked but when i tryed to recover the file in another mapreduce job i got EOFException. I thinked there is 3 possibily:

1- My SequenceFileOutputFormat is wrong and save a corrupted file;

2-My SequenceInputFormt is wrong;

3- The HVPI custom type is wrong.

 

The code to save a  Sequence file:
 
public static void main(String[] args) throws Exception {
 
//判断输入参数
if (args.length != 2) {         
   System.err.println("Usage: <input path> <output path>");     
   System.exit(-1);  
}       
Configuration conf = new Configuration();
 
//设置Master URL
 
conf.set("fs.defaultFS","hdfs://evoido:9000");
Job job = new Job(conf, "MRVideoReader");
job.setJarByClass(MRVideoReader.class);
 
Path in = new Path(args[0]);
Path out = new Path(args[1]);
//reduce task zero job is only map 
job.setJobName("Sequence Writer Test");
job.setNumReduceTasks(0);
job.setInputFormatClass(VideoInputFormat.class);
job.setMapperClass(MRVideoReaderMapper.class);
 
job.setMapOutputKeyClass(Text.class);
//job.setMapOutputValueClass(Text.class);
job.setMapOutputValueClass(ImageWritable.class);
job.setReducerClass(MRVideoReaderReducer.class);
 
job.setOutputFormatClass(SequenceFileOutputFormat.class);
 
job.setJarByClass(MRVideoReader.class);
 
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(ImageWritable.class);
//job.setOutputValueClass(Text.class);
 
 
FileInputFormat.addInputPath(job,in);
 
 
SequenceFileOutputFormat.setOutputPath(job, out);
 
 
System.exit(job.waitForCompletion(true)?0:1);
 
}
 
The code to read from HDFS:
 
public static void main(String[] args) throws Exception {
 
//判断输入参数
if (args.length != 2) {         
   System.err.println("Usage: <input path> <output path>");     
   System.exit(-1);  
}       
Configuration conf = new Configuration();
 
//设置Master URL
 
conf.set("fs.defaultFS","hdfs://evoido:9000");
Job job = new Job(conf, "MRVideoReader");
job.setJarByClass(MRVideoReader.class);
 
Path in = new Path(args[0]);
Path out = new Path(args[1]);
//reduce task zero job is only map 
job.setJobName("Sequence Writer Test");
job.setNumReduceTasks(0);
job.setInputFormatClass(SequenceFileInputFormat.class);
 
job.setMapperClass(MRVideoReaderMapper.class);
 
job.setMapOutputKeyClass(Text.class);
//job.setMapOutputValueClass(Text.class);
job.setMapOutputValueClass(ImageWritable.class);
job.setReducerClass(MRVideoReaderReducer.class);
 
 
job.setOutputFormatClass(TextOutputFormat.class);
job.setJarByClass(MRVideoReader.class);
 
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(ImageWritable.class);
//job.setOutputValueClass(Text.class);
 
SequenceFileInputFormat.addInputPath(job, in);
 
FileOutputFormat.setOutputPath(job, out);
 
 
 
System.exit(job.waitForCompletion(true)?0:1);
}
 
This is the code's piece from the custom type. 
Maybe the readFilds or write methods are incorrect.
 
public void readFields(DataInput in) throws IOException {
int len = WritableUtils.readVInt(in);
System.out.println("Valor de len:"+len);
byte[] temp = new byte[len];
in.readFully(temp, 0, len);
ByteArrayInputStream byteStream = new ByteArrayInputStream(temp); // 输入流;
bufferedImage = ImageIO.read(byteStream); // 从输入流中,读取图片存入image中,而这里in可以为ByteArrayInputStream();
}
 
 
public void write(DataOutput out) throws IOException {
 
 
ByteArrayOutputStream byteOutputStream = new ByteArrayOutputStream();
ImageIO.write( bufferedImage, "png", byteOutputStream );
WritableUtils.writeVInt(out, byteOutputStream.size());
 
}

 

1 ACCEPTED SOLUTION

avatar
Explorer

I founded the error. The HVPI custom type was wrong. I just put out.write(byteOutputStream.toByteArray()); on write method and now it's work.

View solution in original post

2 REPLIES 2

avatar
Explorer

I founded the error. The HVPI custom type was wrong. It was in the write method.

avatar
Explorer

I founded the error. The HVPI custom type was wrong. I just put out.write(byteOutputStream.toByteArray()); on write method and now it's work.