question Re: Nifi: ExecuteSQL returns multiple rows of data, but only one entry is written to MongoDB database in Archives of Support Questions (Read Only)

Nifi: ExecuteSQL returns multiple rows of data, but only one entry is written to MongoDB database

crackerbarrel — Fri, 30 Sep 2016 06:13:47 GMT

testingsqlflow.jpgI have a query that I execute using ExecuteSQL processor that returns 900 rows of data. It Output is in AVRO format which I convert to JSON (one JSON document per row of data) using ConvertAVROtoJSON processor. The output is then sent to a PutMongo processor. But, only the first row from this processor is written to MongoDB. The others are not written and I get get no errors. Any suggestion on how to make sure all rows are written into MongoDB?

I have attached a diagram of the Nifi flow.

Thanks!

Re: Nifi: ExecuteSQL returns multiple rows of data, but only one entry is written to MongoDB database

bhagan — Fri, 30 Sep 2016 21:03:37 GMT

Are you sure that only the first record was written? The NiFi doc says ConvertAvrotoJson converts to a single JSON object.

Re: Nifi: ExecuteSQL returns multiple rows of data, but only one entry is written to MongoDB database

crackerbarrel — Fri, 30 Sep 2016 21:34:41 GMT

Yes, the query returns over 1300 rows. The ConverttoJSON writes both to MongoDB and to a file (using PutFile). The file gets all the rows - one json document per line. But, MongoDB gets only one record which is the first one.

Re: Nifi: ExecuteSQL returns multiple rows of data, but only one entry is written to MongoDB database

bhagan — Fri, 30 Sep 2016 23:09:52 GMT

Try replacing your ConvertAvrotoJSON with a SplitAvro processor. So try a flow like this:

ExecuteSQL > SplitAvro > ConvertAvrotoJSON > PutMongo

Re: Nifi: ExecuteSQL returns multiple rows of data, but only one entry is written to MongoDB database

crackerbarrel — Sat, 01 Oct 2016 01:08:21 GMT

That worked great! Thank you very much! My collection now has over 1300 records as expected.

Re: Nifi: ExecuteSQL returns multiple rows of data, but only one entry is written to MongoDB database

bhagan — Sat, 01 Oct 2016 01:40:36 GMT

Sweet. Glad I could help.