Member since
05-08-2023
12
Posts
0
Kudos Received
0
Solutions
06-16-2023
05:39 AM
@nuxeo-nifi Wanted to first make some suggestions to help us better respond: Include a screen shot of your entire flow Include as much detail as possible about how certain parts are completed. For example: how is the CSV processed. Indicate what you have tried or what you see "toward the end of processing" including details of what you expect. For Example: a single update statement w/ fail and success counts, or insert failures into 1 table and errors into another. Not knowing this, we have to make some assumptions that could possibly result in providing an inaccurate solution or turn the post into long drawn out dialouge, versus simple question, and direct answer/solution. Making those assumptions, I could assume at the bottom of your flow, you have a success and failure relationship. One suggestion would be to use (MergeRecord/MergeContent) to obtain the counts, then maybe replaceText to fabricate the counts into correct shape flowfile and route to an ExecuteSQL processor to execute your SQL statements. Another alternative solution could be to send errors and success to separate ExecuteSQL processors in a way that for each flowfile it just executes a SQL statement that increments the existing count. This would save the need to merge and get totals. Maybe like these in each ExecuteSQL: UPDATE table SET success = success +1 WHERE tablename ='something'
UPDATE table SET errors = errors +1 WHERE tablename ='something'
... View more
06-12-2023
10:10 PM
1 Kudo
@nuxeo-nifi, here is the link to the recording: https://youtu.be/8cZJ9CyLYyI.
Hope this helps.
... View more
05-18-2023
11:28 PM
the input files are csv
... View more
05-09-2023
11:40 PM
@nuxeo-nifi, What I would try to implement as a quick solution is: 1. Configure your ValidateRecord (or even maybe try ValidateCSV) so it identifies when your Records from your CSV are not valid. 2. From ValidateRecord, you have 3 possible queues: failure --> which you might want connect to an alert system, like PutEmail for example. valid --> which you might want to connect to your further processing. invalid --> what you are actually looking for :). Here, you can use an InvokeHTTP to call NiFi's REST API and stop your ValidateRecord Processor. In this way, if a single message was rejected, your entire flow will be stopped... this is actually not the best way to do things but if this is your project requirement, this is what you should do. 2a. From ValidateCSV, you have 2 possible queues: valid --> which you might want to connect to your further processing. invalid --> what you are actually looking for :). Here, you can use an InvokeHTTP to call NiFi's REST API and stop your ValidateCSV Processor. In this way, if a single message was rejected, your entire flow will be stopped... this is actually not the best way to do things but if this is your project requirement, this is what you should do. 3. If you are using this Flow in a so to say Streaming mode (you get files every second), you should modify ValidateRecord to run every 5 seconds or every 2 seconds (or something like that) so you have time to stop your processor using InvokeHTTP. If you leave it by default on Run Schedule 0 sec, you will process some additional messages before being able to stop your processor. Documentation: NiFi Rest API: https://nifi.apache.org/docs/nifi-docs/rest-api/index.html NiFi ValidateRecord: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.ValidateRecord/index.html NiFi InvokeHTTP: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.InvokeHTTP/index.html NiFi ValidateCSV: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.20.0/org.apache.nifi.processors.standard.ValidateCsv/index.html How To ValidateRecord: https://community.cloudera.com/t5/Community-Articles/Using-Apache-NiFi-to-Validate-that-Records-Adhere-to-a/ta-p/247299
... View more
05-09-2023
07:00 AM
@nuxeo-nifi, the processors you are referring to do not belong to any NiFi Version (Cloudera or Open-Source), meaning that they were built in house, specially for you and your project. In this case, you would need to speak to those who have developed those processors and identify the application logic. Once you have that, you can use PutMail to send email notifications and InvokeHTTP to do the other actions. I assume that from your processors you have has a failed connection queue, which might be linked to an PutMail Processor, in which you define whatever you want to be send as notification In case of no failures, you can link the success queue out of your nuxeo processor and into InvokeHTTP and perform the call you require. For that, make sure that all your certificates are in place and allow connection between the systems. Otherwise, you won't be able to use InvokeHTTP and you would have to find another solution, like a script.
... View more