I am bumping this question in hopes someone might know of a better solution  ... View more

A couple of reasons: At the base level, operations in the UI all use the REST API - if you use chrome's devtools and go to the networking tab, you can see the exact REST API route used for every click you do on the canvas. If you'd ever want to automate something you know you can manually do in the UI, using this trick you can perfectly replicate it with with RESTful calls. In the same vain, the REST API has far more routes/operations available. The CLI tools seems to have around ~70 operations, whilst the rest api seems to have over 200. The CLI commands are part of the nifi toolkit. Whilst the toolkit does get updated, I believe it does not get the same level of attention as nifi's main codebase and as such it'd be better not to rely on it completely. This is not to say you can't use the CLI tool - rather, just my opinion on the matter and some insight for how my team writes automations on nifi 🙂 ... View more

Hey @syntax_ , Here is the documentation on NIFI's toolkit CLI commands. It seems there are commands for "pg-stop" and "pg-start", which I believe will allow you to start/stop and manage specific flows inside your canvas by using their IDs.   Personally, I would recommend you use NIFI's REST API for automations rather than CLI commands.   Hope this helps 🙂 ... View more

Hello Miguel,   There are probably ways to optimize tha amount of processors you use, but as a start I would try and monitor where the slowness comes from. Is there any backpressure in the flow (files queueing up before a processor)? Are you sending requests at the same rate? if so does that mean those 6000 records/second you lost translate to 5xx responses from nifi to the origin service?   In terms of optimizing, perhaps you can transform null values with your initial JoltTransformJson. it sounds like you use a lot of ReplaceText processors too - perhaps UpdateRecord would be of more use for updating your fields. I'd recommend looking at QueryRecord for your validation/routing instead of using UpdateAttribute -> RouteOnAttribute (if applicable and you route based on the json fields). ... View more

Hello Mahendra,   Have you tried sending an update request that includes all the variables except the one you wish to delete? If that doesn't work I'll run some tests next time I'm at office and post an update. ... View more

In the end, I never found a perfect solution and just opted to use attributes and RouteOnAttribute. If you find ValidateXML can actually verify all the checks you need, it shouldn't be too bad using a ConvertRecord processor and transforming JSON to XML for the validation (or perhaps just straight up using XMLs instead of jsons if that fits your use case) ... View more

I think I may not have understood your requirement correctly so please correct me if I'm wrong. Currently, you have managed to transform your data into the format of:   { "ret_code":0, "ret_msg":"OK", "ext_code":"", "ext_info":"", "result":[ { "id":"184310970522", "symbol":"BTCUSDT", "price":19241.5, "qty":1.896, "side":"Buy", "time":"2022-10-11T12:26:21.000Z", "trade_time_ms":1665491181666, "is_block_trade":false }, { "id":"184310967802", "symbol":"BTCUSDT", "price":19241, "qty":0.002, "side":"Sell", "time":"2022-10-11T12:26:21.000Z", "trade_time_ms":1665491181604, "is_block_trade":false } ], "time_now":"1665491183.183636" }    Where you have a single json record with a field "result" that is an array of results. I understood your need as transforming this single json with two results into two separate records with 1 result each where the rest of their fields are identical (ret_code,ret_msg, ext_code, etc.), I.E:   [ { "ret_code":0, "ret_msg":"OK", "ext_code":"", "ext_info":"", "result":{ "id":"184310970522", "symbol":"BTCUSDT", "price":19241.5, "qty":1.896, "side":"Buy", "time":"2022-10-11T12:26:21.000Z", "trade_time_ms":1665491181666, "is_block_trade":false }, "time_now":"1665491183.183636" }, { "ret_code":0, "ret_msg":"OK", "ext_code":"", "ext_info":"", "result":{ "id":"184310967802", "symbol":"BTCUSDT", "price":19241, "qty":0.002, "side":"Sell", "time":"2022-10-11T12:26:21.000Z", "trade_time_ms":1665491181604, "is_block_trade":false }, "time_now":"1665491183.183636" } ]   If this is not what you need, I'd like you to post an example for what your data needs to look like before inserting to your DB. I can help with transforming it to the right format only if I'm sure of what is required 🙂 ... View more

I'm not too sure what you mean, so let me see if I got it right. You managed to get your data into the format of:   { "ret_code":0, "ret_msg":"OK", "ext_code":"", "ext_info":"", "result":[ { "id":"184310970522", "symbol":"BTCUSDT", "price":19241.5, "qty":1.896, "side":"Buy", "time":"2022-10-11T12:26:21.000Z", "trade_time_ms":1665491181666, "is_block_trade":false }, { "id":"184310967802", "symbol":"BTCUSDT", "price":19241, "qty":0.002, "side":"Sell", "time":"2022-10-11T12:26:21.000Z", "trade_time_ms":1665491181604, "is_block_trade":false } ], "time_now":"1665491183.183636" }   And now you want to insert this data into your DB. Then, you mentioned how you need to split the data before you can write, which I assumed meant you want to transform the above example into:   [ { "ret_code":0, "ret_msg":"OK", "ext_code":"", "ext_info":"", "result":{ "id":"184310970522", "symbol":"BTCUSDT", "price":19241.5, "qty":1.896, "side":"Buy", "time":"2022-10-11T12:26:21.000Z", "trade_time_ms":1665491181666, "is_block_trade":false }, "time_now":"1665491183.183636" }, { "ret_code":0, "ret_msg":"OK", "ext_code":"", "ext_info":"", "result":{ "id":"184310967802", "symbol":"BTCUSDT", "price":19241, "qty":0.002, "side":"Sell", "time":"2022-10-11T12:26:21.000Z", "trade_time_ms":1665491181604, "is_block_trade":false }, "time_now":"1665491183.183636" } ]   Which is an array of two records, both of which have identical ret_code, ret_msg, ext_code, ext_info & time_now fields, except now instead of the result field being an array of two results, the two records each have only one result inside of them.   Could you give a better example of what the data you want to insert into your DB should look like? I can help with figuring out how to transform it into a valid format but first I need to have a better understanding of how your data looks like currently and what it needs to turn into 🙂 ... View more