Member since
10-21-2016
5
Posts
0
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3210 | 10-22-2016 09:00 PM |
10-22-2016
09:00 PM
I founded the error. The HVPI custom type was wrong. I just put out.write(byteOutputStream.toByteArray()); on write method and now it's work.
... View more
10-22-2016
08:40 PM
I founded the error. The HVPI custom type was wrong. It was in the write method.
... View more
10-21-2016
04:00 AM
Hello. I tryed to process image on hadoop and i using HVPI (https://github.com/xmpy/hvpi). It's open source. I using it to extract the frames from video but it not provide a output format to save the frames on HDFS. So i tryed to use the SequencefileOutputFormat to save the frames on HDFS. I readed a book and make some changes to the code work on hadoop 2.7.1. Apparently it's worked but when i tryed to recover the file in another mapreduce job i got EOFException. I thinked there is 3 possibily: 1- My SequenceFileOutputFormat is wrong and save a corrupted file; 2-My SequenceInputFormt is wrong; 3- The HVPI custom type is wrong. The code to save a Sequence file: public static void main(String[] args) throws Exception { //判断输入参数 if (args.length != 2) { System.err.println("Usage: <input path> <output path>"); System.exit(-1); } Configuration conf = new Configuration(); //设置Master URL conf.set("fs.defaultFS","hdfs://evoido:9000"); Job job = new Job(conf, "MRVideoReader"); job.setJarByClass(MRVideoReader.class); Path in = new Path(args[0]); Path out = new Path(args[1]); //reduce task zero job is only map job.setJobName("Sequence Writer Test"); job.setNumReduceTasks(0); job.setInputFormatClass(VideoInputFormat.class); job.setMapperClass(MRVideoReaderMapper.class); job.setMapOutputKeyClass(Text.class); //job.setMapOutputValueClass(Text.class); job.setMapOutputValueClass(ImageWritable.class); job.setReducerClass(MRVideoReaderReducer.class); job.setOutputFormatClass(SequenceFileOutputFormat.class); job.setJarByClass(MRVideoReader.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(ImageWritable.class); //job.setOutputValueClass(Text.class); FileInputFormat.addInputPath(job,in); SequenceFileOutputFormat.setOutputPath(job, out); System.exit(job.waitForCompletion(true)?0:1); } The code to read from HDFS: public static void main(String[] args) throws Exception { //判断输入参数 if (args.length != 2) { System.err.println("Usage: <input path> <output path>"); System.exit(-1); } Configuration conf = new Configuration(); //设置Master URL conf.set("fs.defaultFS","hdfs://evoido:9000"); Job job = new Job(conf, "MRVideoReader"); job.setJarByClass(MRVideoReader.class); Path in = new Path(args[0]); Path out = new Path(args[1]); //reduce task zero job is only map job.setJobName("Sequence Writer Test"); job.setNumReduceTasks(0); job.setInputFormatClass(SequenceFileInputFormat.class); job.setMapperClass(MRVideoReaderMapper.class); job.setMapOutputKeyClass(Text.class); //job.setMapOutputValueClass(Text.class); job.setMapOutputValueClass(ImageWritable.class); job.setReducerClass(MRVideoReaderReducer.class); job.setOutputFormatClass(TextOutputFormat.class); job.setJarByClass(MRVideoReader.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(ImageWritable.class); //job.setOutputValueClass(Text.class); SequenceFileInputFormat.addInputPath(job, in); FileOutputFormat.setOutputPath(job, out); System.exit(job.waitForCompletion(true)?0:1); } This is the code's piece from the custom type. Maybe the readFilds or write methods are incorrect. public void readFields(DataInput in) throws IOException { int len = WritableUtils.readVInt(in); System.out.println("Valor de len:"+len); byte[] temp = new byte[len]; in.readFully(temp, 0, len); ByteArrayInputStream byteStream = new ByteArrayInputStream(temp); // 输入流; bufferedImage = ImageIO.read(byteStream); // 从输入流中,读取图片存入image中,而这里in可以为ByteArrayInputStream(); } public void write(DataOutput out) throws IOException { ByteArrayOutputStream byteOutputStream = new ByteArrayOutputStream(); ImageIO.write( bufferedImage, "png", byteOutputStream ); WritableUtils.writeVInt(out, byteOutputStream.size()); }
... View more
Labels:
- Labels:
-
Apache Hadoop
-
HDFS
-
MapReduce