InferAvroSchema should have Schema Output Destination set to "flowfile-attribute", the outgoing flow file should contain the CSV data and an attribute called "inferred.avro.schema" which contains the schema to use. Then in ConvertCSVToAvro you can set the Record Schema property to "${inferred.avro.schema}" which will cause it to use the inferred schema for conversion. Since you are entering the CSV Header Definition manually, you may find it more helpful to create an Avro schema manually and use ConvertRecord instead of InferAvroSchema -> ConvertCSVToAvro. If you don't know the datatypes of the columns and are thus relying on InferAvroSchema to do that for you, you could still use ConvertRecord instead of ConvertCSVToAvro. ... View more

ListHDFS emits empty (0-byte) flow files that have attributes (such as filename and path, see the doc for details) set on them. In this case FetchHDFS is running way more slowly than ListHDFS (it takes longer to retrieve the file than to list that it's there), which is why you get the backup. Also setting Max Size as a backpressure trigger won't work here since they are 0-byte files. Try setting Max number of Objects for backpressure instead. ... View more

What version of the Hive driver are you using? I'm not sure there is a version of the Hive driver available that supports all the JDBC API calls made by PutDatabaseRecord, such as executeBatch(). Also since the Hive JDBC driver auto-commits after each operation, PutDatabaseRecord + Hive would not be any more performant than using PutHiveQL. In an upcoming version of NiFi/HDF (for Hive 3.0), you should be able to use PutHive3Streaming to do what you want. ... View more

The variables are in a ComponentVariableRegistry which is pretty well-hidden under the NiFi API. Usually you get at variables by evaluating Expression Language in the context of a processor property. In this case I set a Process Group variable called "myAttr" to "myValue", then I configured ExecuteScript like so: Note that I created a user-defined property "myProp" whose value is an expression language construct containing the PG variable name. When you call evaluateAttributeExpressions() on that property (see the script) it will resolve the value of "myAttr" and return that, you can verify that an outgoing flow file would now have "myFlowFileAttr" set to "myValue". ... View more