Support Questions
Find answers, ask questions, and share your expertise

Use NiFi to Insert data into MongoDB array

Use NiFi to Insert data into MongoDB array

New Contributor

Newbie question: I'm working on an ingest process that accepts a group of files, routes one to MongoDB and inserts it. This creates a basic structure for the document. Fields included are:_id - (ObjectId automatically added by MongoDB, id, author, submission_type, submission_name, abstract, data (array). The process assumes a json file for the basic structure and attachments for the submission. I am able to use ListFile, FetchFile, and RouteOnAttribute files to grab the files. I use RouteOnAttribute to separate the json structure file from the attachment files. The json file is inserted into MongoDB using PutMongoRecord. Everything is inserted fine.

 

The issue comes with the attachment files. They are routed to ExtractMediaMetadata to grab metadata attributes. Using processors SplitJson, EvaluateJsonPath, and AttributesToJson, I gather all the attributes to insert in a MongoDB record. Now I want to match the filename attributes from the attachment files to the mongoDB document I just inserted, matching on db.collection.data which is an array.

Example of JSON MongoDB insert:

_id: 5e0b7f98f81e7cdb40f974a4
group_id:1

author: "John Q Sample"

submission_type: "Music"

submission_name"Classical Compilation"

abstract :"This submission.... ..... .... more abstract info"

dataArray

0"File1.docx"

1"File2.docx"

2"File3.ppt"

3"File4.xls"

 

The filename attributes of the attachment files will match the filenames stored in the data array that was inserted into MongoDB. Is there a way that I can use NiFi to query the MongoDB document (db.collection.data, look at array and match the filenames) then insert the json attributes from the attachment files into the data array? Any advice would be appreciated!