Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Multiple JSON to JSON Array Output file

Multiple JSON to JSON Array Output file

Explorer

Hello All, 

 

I am exporting data from Mongo DB. In mongo db collection i have round 2000+ objects. These are read using GetMongo. Further processed by EvaluateJSON and then SplitJSON. 

 

Now post that i want to write all the JSON as single JSON array in a single JSON file. but not able to achieve it. 

 

Tried calling UpdateAttribute updating output file name and then call to Putfile. but putfile is writing only one record and then fails with "duplicate file or same file exist " error. 

1. Can anyone direct me on how to get these multiple JSON in to a single JSON array. 1. 

2. Fetch only 200 records by GetMongo so that each file is only 200 JSON Array. 

 

Please guide 

 

Thxs

Rups

 

 

  

2 REPLIES 2
Highlighted

Re: Multiple JSON to JSON Array Output file

@Rups Your GetMongo should have a Limit, Batch Size, and Results Per FlowFile.   Investigate using these to get the # of results you want per FlowFile.    If you do not see these configurations investigate nifi 1.9 or greater.  

 

I would recommend NOT using SplitJson to split results.

 

For putFile you need to update the filename to something unique per FlowFile to avoid the duplicate filename.  For example something like:

 

filename = ${filename:append(now():format('yyyy-mm-dd hh:mm:ss'))}

 

or use the FlowFile's Filename:

 

filename = ${Filename}

 


 


If this answer resolves your issue or allows you to move forward, please choose to ACCEPT this solution and close this topic. If you have further dialogue on this topic please comment here or feel free to private message me. If you have new questions related to your Use Case please create separate topic and feel free to tag me in your post.  


 


Thanks,



Steven

Highlighted

Re: Multiple JSON to JSON Array Output file

Explorer

Above solution has already been tried. Here with this I get individual file with one JSON in each. 

Whereas i need single file with multiple JSON in that. Is that possible ?

Don't have an account?
Coming from Hortonworks? Activate your account here