One way would be to define the schema ahead of time in one of the schema registries, and then have your CSVReader's Schema Access Strategy set to "Schema Name" so that it uses the schema from the registry, and then tell it to ignore the first line of the CSV. The downside is you have to define the schema rather than just using the column headers. Besides that, the next easiest option would probably be to use ExecuteScript to write a simple script that reads the first line and converts the spaces in the column names to underscores, and then wrote it back out converted along with all the other unmodified lines. It is possible there might be a way to do it with ReplaceText, but I'm not exactly sure how to modify only the first line. ... View more

Variable registry is not the same thing as flow file attributes. What you are setting here are flow file attributes with the topic names, which will now be passed along with that specific flow file and accessible through expression language only when that particular flow file is involved. There is no programmatic access to the variable registry from a processor or ExecuteScript, it would have to be done via NiFi's REST API. ... View more

Increasing the back-pressure threshold will allow you to send more data to the input port, but it won't help make PublishKafka work. Where does the 1.2GB file come from and how does it get sent to the input port? I'm wondering if you can create multiple smaller files, instead of one huge file, and see if that helps at all. ... View more

The Run Schedule is the schedule of when the NiFi framework will execute a processor. The default of timer driver 0 seconds means to execute as fast as possible when there is data available in the incoming queue, if no data is there then it doesn't execute. The rate of the data depends on what the processor does during one execution... for example, lets say a queue has 100 flow files in it and you set the processor to run every 5 minutes. Some processors may grab a batch of files during one execution, so even tough the processor executes once, it may grab 50 of those flow files. It also depends if your flows files have multiple logical messages in the content. If you have 1 record per flow file, and if the processor only grabs 1 flow file at a time (most only take one at a time), then the run schedule does control the rate. You can look at ControlRate processor as well. ... View more

You can bulk manipulate records using UpdateRecord and Record Path: https://nifi.apache.org/docs/nifi-docs/html/record-path-guide.html https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.6.0/org.apache.nifi.processors.standard.UpdateRecord/index.html Changing the name of the field is a little tricky because you are reading based on a schema, so the field name is part of the schema, but it can be done if you define both field names in the schema and have the second field default to null, then you can move the value from the first field to the second field. Regarding splitting a record in the binary and metadata... One thing to understand is that even though your Avro data file has a schema embedded in it, you don't have to use this schema for reading/writing in NiFi. Imagine this scenario... Embedded schema: { "name": "foo", "namespace": "nifi", "type": "record", "fields": [ { "name": "field1", "type": "string" }, { "name": "field2", "type": "bytes" } ] } You could define two other schemas and put them in NiFi's schema registry... Metadata only: { "name": "foo", "namespace": "nifi", "type": "record", "fields": [ { "name": "field1", "type": "string" } ] } Binary Only: { "name": "foo", "namespace": "nifi", "type": "record", "fields": [ { "name": "field2", "type": "bytes" } ] } Then you can have a ConvertRecord that has the AvroReader using embedded schema, and the AvroWriter using a schema from the schema registry. So on the way out it would rewrite the Avro with only the fields from the writer schema. So if you sent your original data to two different ConvertRecord processors, one for metadata only and one for binary only, then can essentially select out the parts for each one. ConvertRecord is preferred over other convert processors. How the schema is accessed is based on Schema Access Strategy in the services. If a record reader says "Schema By Name" then it requires a schema registry to look up by name. If the Avro reader has "Embedded Schema" select then you shouldn't need to use a schema registry because it will use the schema embedded in the Avro file. ... View more