Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Why we use IntWritable instead of Int? Why we use LongWritable instead of Long?

avatar
Explorer
 
1 REPLY 1

avatar
Master Mentor

@Sheetal Sharma


LongWritable is the WritableComparable for longs, Similarly IntWritable is a WritableComparable for ints.

These interfaces [1] & [2] are all necessary for Hadoop/MapReduce, as the Comparable interface is used for comparing when the reducer sorts the keys, and Writable can write the result to the local disk. It does not use the java Serializable because java Serializable is too big or too heavy for hadoop, Writable can serializable the hadoop Object in a very light way.



[1] https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/io/LongWritable.html
[2] https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/io/IntWritable.html#IntWritable()


"Comparable" is the interface whose abstract methods give us the flexibility to compare two objects.


"Writable" is meant for writing the data to local disk and it's a serialization format. One can implement own Writables in Hadoop. Java’s serialization is too bulky and slow on the system. That’s why Hadoop community had put Writable in place.


"WritableComparable" is a combination of the above two interfaces.


"int" is a primitive type so it cannot be used as key-value. Integer is the wrapper class around it. So I’ll correct your question that what is the difference between Integer and IntWritable?


"IntWritable" is the Hadoop variant of Integer which has been optimized for serialization in the Hadoop environment. An integer would use the default Java Serialization which is very costly in Hadoop environment.

.