Support Questions
Find answers, ask questions, and share your expertise

I want to insert data into the database without converting data into json

Hi....I have data in various kinds of files and i want to insert them into database but i don't want it to convert them first into JSON and then to SQL as it is very time consuming.Is there any way so that i can put my data into database from different kinds of files like CSV,Avro etc directly?

1 ACCEPTED SOLUTION

Super Guru

For CSV files, if you know the number and type of column values, you can use SplitText (to get one row per flow file) followed by ExtractText, supplying a regular expression to get the column values out into flow file attributes. Then you can use ReplaceText to manually enter a SQL INSERT statement (using NiFi Expression Language to access the attributes). For other formats like Avro, as we don't currently have a ConvertAvroToSQL processor, you'd have to convert them for now.

Work is underway for a generic system of type conversions, such that you could specify Avro as your input format and perhaps "SQL INSERT" as your output format, thereby effectively making the generic processor work like a ConvertAvroToSQL processor.

View solution in original post

1 REPLY 1

Super Guru

For CSV files, if you know the number and type of column values, you can use SplitText (to get one row per flow file) followed by ExtractText, supplying a regular expression to get the column values out into flow file attributes. Then you can use ReplaceText to manually enter a SQL INSERT statement (using NiFi Expression Language to access the attributes). For other formats like Avro, as we don't currently have a ConvertAvroToSQL processor, you'd have to convert them for now.

Work is underway for a generic system of type conversions, such that you could specify Avro as your input format and perhaps "SQL INSERT" as your output format, thereby effectively making the generic processor work like a ConvertAvroToSQL processor.

; ;