Member since
10-19-2018
3
Posts
0
Kudos Received
0
Solutions
01-28-2019
10:29 PM
HI, I am working on a requirement to load a JSON file into a Database. Before doing the ingestion, I have to validate the Json response against the schema. I read about Validate Record Processor, but that does not fit in my requirement because I want the error records in a flow file. With Validate Record Processor, the error records are moved to logs. I am trying to use RouteOnAttribute Processor and use an expression language to validate Date Columns in Particular. Any Suggestions? Can we achieve this using EL?
... View more
- Tags:
- Data Ingestion & Streaming
- expression-language
- nifi-controller-service
- nifi-processor
- nifi-templates
- validation
Labels:
01-17-2019
06:46 AM
HI, I am working on a requirement to load a JSON file into a Database. Before doing the ingestion, I have to validate the Json response against the schema. I read about Validate Record Processor, but that does not fit in my requirement because I want the error records in a flow file. With Validate Record Processor, the error records are moved to logs. I am trying to use RouteOnAttribute Processor and use an expression language to validate Date Columns in Particular. Any Suggestions? Can we achieve this using EL?
... View more
- Tags:
- Data Ingestion & Streaming
- expression-language
- nifi-processor
- nifi-state
- nifi-templates
- validate-schema
Labels:
10-19-2018
04:50 AM
Hi All, My requirement is to load a csv file into a database using NiFi. The current approach (which is inefficient) is SplitText--> ExtractText-->UpdateAttribute-->ReplaceText-->PutSQL. I am looking for a solution that lets me do a batch commit and not individual INSERTs for each FlowFile. I have seen a couple of responses online like - use PutDatabaseRecord Processor or ConvertJSONToSQL Processor. But, I have to invoke UDFs in the Insert query to transform the data before inserting into the table. I do not think I will be able to use the above mentioned approaches if UDFs are involved. Is there a way to Begin and commit the Insert Statements before passing them to PutSQL ? Can we invoke any SQL utility to achieve that? I would really appreciate your help.
... View more
Labels: