question Re: how to configure mergecontent processor in Archives of Support Questions (Read Only)

how to configure mergecontent processor

mark_hadoop — Wed, 05 Sep 2018 16:42:14 GMT

Hi All,

I have json data in multiple small files (some times only one line in a file).

I want to merge all small files in to single large file.

I am getting a large file in an unexpected format.

ex:

file 1:

{"code"="1", "color"="green"}
{"code"="2", "color"="blue"}
{"code"="3", "color"="orange"}

file 2:

{"code"="4", "color"="yellow"}
{"code"="5", "color"="red"}

I am getting the below output after using MergeContent

{"code"="1", "color"="green"}
{"code"="2", "color"="blue"}
{"code"="3", "color"="orange"}{"code"="4", "color"="yellow"}
{"code"="5", "color"="red"}

Expected output

{"code"="1", "color"="green"}
{"code"="2", "color"="blue"}
{"code"="3", "color"="orange"}
{"code"="4", "color"="yellow"}
{"code"="5", "color"="red"}

bbende — Thu, 06 Sep 2018 02:31:36 GMT

I answered this on stackoverflow:

mark_hadoop — Thu, 06 Sep 2018 10:35:22 GMT

I have checked my data, it has no new blank spaces, but was arriving like batches.

I am merging the files and appending using puthdfs.

When I use the configurations you suggested, some times I am getting a new blank line at the beginning of the file which is appended using puthdfs.

Can you please help me how to avoid the blank line at the beginning of the file, also the file is big (1GB).