https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-create-a-hadoop-custom-inputformat-fileinputformat/m-p/175445#M75432 <P>thanks <A rel="user" href="https://community.cloudera.com/users/44302/kgautam.html" nodeid="44302">@kgautam</A> this is really helpful.</P> Thu, 08 Mar 2018 09:04:59 GMT melvinmendoza 2018-03-08T09:04:59Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-create-a-hadoop-custom-inputformat-fileinputformat/m-p/175443#M75430 <P>Any knows or have a tutorial?</P> Wed, 07 Mar 2018 15:50:47 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-create-a-hadoop-custom-inputformat-fileinputformat/m-p/175443#M75430 melvinmendoza 2018-03-07T15:50:47Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-create-a-hadoop-custom-inputformat-fileinputformat/m-p/175444#M75431 <P>Please go through the basic understanding of InputFormats and Record Readers </P><P><A href="http://bytepadding.com/big-data/map-reduce/how-records-are-handled-map-reduce/" target="_blank">http://bytepadding.com/big-data/map-reduce/how-records-are-handled-map-reduce/</A><BR /><A href="http://bytepadding.com/big-data/map-reduce/how-records-are-handled-map-reduce/" target="_blank">http://bytepadding.com/big-data/map-reduce/how-records-are-handled-map-reduce/</A><BR /></P><P>Example of custom Input Formats </P><P><A href="http://bytepadding.com/big-data/spark/combineparquetfileinputformat/" target="_blank">http://bytepadding.com/big-data/spark/combineparquetfileinputformat/</A><BR /><BR /><BR />Few pointers:<BR />1. Start with basic understanding of Splits, InputFormats, Record Readers, File formats and compression.<BR />2. Go through the code of TextInputFormat : <A href="http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.collector/1.1.0-pre7/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.java" target="_blank">http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.collector/1.1.0-pre7/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.java</A></P><P>3. FileInputFormat is the abstract class or the Base class for all input formats go through the basic functionality</P><P><A href="http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.collector/1.1.0-pre7/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java#FileInputFormat" target="_blank">http://grepcode.com/file/repo1.maven.org/maven2/com.ning/metrics.collector/1.1.0-pre7/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java#FileInputFormat</A><BR /><BR />4. Decide upon what is the logical record for your InputFormat and whats the splitting Strateggy, depending on this extend the </P><P>FileInputFormat and override/ implement getSplits() and getRecordReader() methods. <BR /><BR />FileInputFormat important method: <BR /> getSplits() : each task will read one split, what is the start file index and end inex for this split<BR />getRecordReader() : the split being read how bytes needs to be converted into bytes. <BR /><BR /><BR /></P> Wed, 07 Mar 2018 18:43:41 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-create-a-hadoop-custom-inputformat-fileinputformat/m-p/175444#M75431 kgautam 2018-03-07T18:43:41Z https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-create-a-hadoop-custom-inputformat-fileinputformat/m-p/175445#M75432 <P>thanks <A rel="user" href="https://community.cloudera.com/users/44302/kgautam.html" nodeid="44302">@kgautam</A> this is really helpful.</P> Thu, 08 Mar 2018 09:04:59 GMT https://community.cloudera.com/t5/Archives-of-Support-Questions/How-to-create-a-hadoop-custom-inputformat-fileinputformat/m-p/175445#M75432 melvinmendoza 2018-03-08T09:04:59Z