I'm running Flume-1.8 with the spoolDir as flume source and hdfs as a sink. I need to process log files in spoolDir with the lines containing more than 10,000 characters and guarantee the file in the hdfs will contain exactly the same number of lines.
According to the Flume-1.8 User Guide the DEFAULT Maximum number of characters to include in a single event is 2048. If a line exceeds this length, it is truncated, and the remaining characters on the line will appear in a subsequent event.
I need to set this value of deserializer.maxLineLength as bigger as possible to prevent input file's lines truncating.
Is there any limit for the value of deserializer.maxLineLength in Flume-1.8 ?