For CSV files, if you know the number and type of column values, you can use SplitText (to get one row per flow file) followed by ExtractText, supplying a regular expression to get the column values out into flow file attributes. Then you can use ReplaceText to manually enter a SQL INSERT statement (using NiFi Expression Language to access the attributes). For other formats like Avro, as we don't currently have a ConvertAvroToSQL processor, you'd have to convert them for now. Work is underway for a generic system of type conversions, such that you could specify Avro as your input format and perhaps "SQL INSERT" as your output format, thereby effectively making the generic processor work like a ConvertAvroToSQL processor. ... View more

Is your processor in its own NAR, or have you added it to a NiFi NAR (such as the nifi-hive-bundle or nifi-hdfs-bundle)? If the former, have you added the nifi-hadoop-libraries NAR as a parent to your NAR? This will give you access to the Hadoop JARs/classes via a parent classloader. To add this NAR as a parent, add the following to the <dependencies> section in your custom processor's NAR module (not the processor module itself): <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-hadoop-libraries-nar</artifactId> <type>nar</type> </dependency> Can you describe your use case a little more? If your files are already in ORC format, you should be able to use PutHDFS to place them onto the Hadoop cluster. If they are in some other format, you might be able to use some conversion processors (including ConvertAvroToORC) and then PutHDFS to land the resultant ORC files into the cluster. ... View more

According to this, Salesforce requires authentication via OAuth 2.0. Please see this HCC article for a discussion (and related links) on how to achieve this. The question is about the GetHttp processor but should apply to InvokeHttp as well. ... View more

It does convert Avro to ORC, but ironically it does so by converting the Avro to JSON first. We should allow ORC (if prudent, possibly after a ConvertAvroToORC processor) and JSON as valid input formats (perhaps via the mime.type attribute?) to PutHiveStreaming. Do you mind writing up a Jira for this? Please and thank you 🙂 ... View more

NIFI-3413 describes a GetChangeDataCapture processor, which will generate events that correspond to database updates (rows added, tables changed, e.g.). The hope is that it will support MySQL, Oracle, and SQL Server (although the first release of the processor may not). Development of the MySQL CDC support (via its binlog files) is underway. ... View more