@Kaliyug Antagonist, JSON files by definition are splittable (technically), as they are text files. But not always they are logically splittable. There are three scenarios with JSON formatted files: 1. Each file contains multiple JSON objects, having 1 object per line. {"a":"x", "b":"y", "c":"z"}
{........} 2. Each file contains single JSON object on multiple lines. {
"a":"x",
"b":"y",
"c":"z"
}
3. Each file contains multiple JSON objects, each JSON object can be on multiple lines. {
"a":"x1",
"b":"y1",
"c":"z1"
}
{
"a":"x2",
"b":"y2",
"c":"z2"
} In first case, JSON files are logically splittable, so you can use spark sql, M/R, PIG and massage a data as per requirements. In second case, it makes sense to pre-process them by removing "new line" chars and pushing one-line-json-objects into single/multiple files on output (depending on total volumes, RM capacity, containers memory, etc). Third case is similar to second one, just more complex as you will need to identify beginning/end of each object to make separate lines for each. How to combine. Simple M/R job can do the trick. Define new class for non-splittable text input format: import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public class NonSplitTextInputFormat extends TextInputFormat {
@Override
protected boolean isSplitable(FileSystem fs, Path file) {
return false;
}
}
In mapper implementation, override "run" method: Note, for second use case you can just pull a call to "map" method out of loop and avoid redundant condition. Then you can either save the output directly to HDFS, or use reducers, to produce specific number of files (or even single one). This solution performed very well for me for huge data sets.
... View more
