@Dennis Fridlyand, thank you for sharing the mapper and reducer code. I think I've spotted a bug in the reducer that I can help with. protected void map(LongWritable key, BytesWritable value,
Mapper<LongWritable, BytesWritable, Text, Text>.Context context) throws IOException,
InterruptedException {
final String json = new String(value.getBytes(), "UTF-8"); Here, value is an instance of BytesWritable. A BytesWritable is a wrapper over an underlying byte array. That underlying buffer may be reused multiple times to represent different records. The actual length of the data within the buffer that is considered valid is tracked separately using a numeric size field. For more details, please see the source code for BytesWritable. By calling value.getBytes(), the mapper code is accessing the raw underlying buffer. This buffer might still contain trailing data from a previous record. Only the data in the buffer up to the length returned by value.getLength() is truly valid. The recommendation is to switch to using the copyBytes() method, which contains additional logic to copy only the currently valid bytes of the underlying buffer. I recommend making the following change in the mapper code. final String json = new String(value.copyBytes(), "UTF-8"); Would you please try that?
... View more
