Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

how to configure mergecontent processor

avatar
Expert Contributor

Hi All,

I have json data in multiple small files (some times only one line in a file).

I want to merge all small files in to single large file.

I am getting a large file in an unexpected format.

ex:

file 1:
{"code"="1", "color"="green"}
{"code"="2", "color"="blue"}
{"code"="3", "color"="orange"}

file 2:
{"code"="4", "color"="yellow"}
{"code"="5", "color"="red"}
I am getting the below output after using MergeContent
{"code"="1", "color"="green"}
{"code"="2", "color"="blue"}
{"code"="3", "color"="orange"}{"code"="4", "color"="yellow"}
{"code"="5", "color"="red"}
Expected output
{"code"="1", "color"="green"}
{"code"="2", "color"="blue"}
{"code"="3", "color"="orange"}
{"code"="4", "color"="yellow"}
{"code"="5", "color"="red"}
1 ACCEPTED SOLUTION

avatar
Master Guru
2 REPLIES 2

avatar
Master Guru

avatar
Expert Contributor

@Bryan Bende

I have checked my data, it has no new blank spaces, but was arriving like batches.

I am merging the files and appending using puthdfs.

When I use the configurations you suggested, some times I am getting a new blank line at the beginning of the file which is appended using puthdfs.

Can you please help me how to avoid the blank line at the beginning of the file, also the file is big (1GB).