Member since
05-03-2023
1
Post
0
Kudos Received
0
Solutions
05-03-2023
10:43 AM
Hi all, We are trying to configure a flow in Nifi where failed sql queries are saved after some retries to local file for further analysis. I already see the following links related to error handling in nifi https://youtu.be/nrxv65J4Ctg by Steven Koon https://insights.1904labs.com/blog/2020-11-12-creating-an-error-retry-framework-in-nifi-part-2 But those articles do not refer to PutSql processor, I'm not sure if the case apply. The failed query in our scenario using PutSQL processor is related to wrong registry information on origin database, giving error inserting a text value on a type bigint column on destination sql server database. This is an example sql query used for inserts on destination database: INSERT INTO destination (id, name, description, group_id) VALUES (100, 'Bob', 'Example query', '2151') We have a wrong registry on origin database in where the group_id field have value 'null' getting the following exception on nifi logs: org.apache.nifi.processor.exception.ProcessException: Failed to process StandardFlowFileRecord[uuid=8fb0f61a-dda2-4101-befe-24ca18ced5c0,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1683132909199-67, container=default, section=67], offset=36504, length=104],offset=0,name=56638e31-2c44-4506-bac7-a38a2e612301,size=104] due to java.sql.SQLDataException: The value of the sql.args.4.value is 'null', which cannot be converted into the necessary data type at org.apache.nifi.processor.util.pattern.ExceptionHandler.lambda$createOnGroupError$2(ExceptionHandler.java:226) at org.apache.nifi.processor.util.pattern.ExceptionHandler.lambda$createOnError$1(ExceptionHandler.java:179) As the error shows the reason is that the destination group_id field is of type bigint. We need to manage cases like that (registries with wrong data or format where the query fails). I configured a PutFile processor to save the queries that failed in PutSQL processor after some retries, this is the content in the local file created by PutFile processor after running the flow. ----------------------------------------------------------------------------------------------------------- INSERT INTO topology_node_destination (id, natural_key, description, infotecnica_id) VALUES (?, ?, ?, ?) ----------------------------------------------------------------------------------------------------------- This is the part of our flow concerning My questions are the following: Is there a way to save the query with the PutFile or another processor with the real values used by PutSQL processor, in a way to know the exact query that is causing the error ? If that can not be achieved, what is the recommended practice ? Thank you for reading.
... View more
Labels:
- Labels:
-
Apache NiFi