I'm not sure I understand the question... There is a controller service called HBaseClientService which would be the connection to HBase, and then your processor can declare a property like: static final PropertyDescriptor HBASE_CLIENT_SERVICE = new PropertyDescriptor.Builder() .name("HBase Client Service") .description("Specifies the Controller Service to use for accessing HBase.") .required(true) .identifiesControllerService(HBaseClientService.class) .build(); Then in your onTrigger method you get the service by doing: HBaseClientService clientService = context.getProperty(HBASE_CLIENT_SERVICE) .asControllerService(HBaseClientService.class); You can look at how the existing HBase processors work: https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-hbase-bundle/nifi-hbase-processors ... View more

I believe event-driven is still somewhat experimental... For processors that have in an incoming queue, generally you schedule them with timer-driver and a run schedule of 0 which means run all the time, and then the framework only runs them when the queue has data in it. So if your queue is always empty it won't be using any CPU cycles. Is there something about your processor that would cause the above approach to not work? ... View more

In Apache NiFi 1.2, there are is PublishKafkaRecord_0_10 which uses a RecordReader service for reading incoming flow files as records, and a RecordWriter service for serializing records to bytes that will be published to a Kafka topic. Each RecordReader and RecordWriter can be configured with a strategy for accessing a schema, one strategy is to get a schema by name from a schema registry. If using a schema registry, you can configure a schema registry service. Apache NiFi provides AvroSchemaRegistry as a local schema registry that can be used within NiFi, or a HortonworksSchemaRegistry service which can be used with an external Hortonworks schema registry. In addition, anyone could implement their own schema registry service to talk to any other schema registry or backing storage mechanism. ... View more

I agree with what Matt said above and I had been working on a template to achieve this before I saw his answer so figured I would post it anyway... I made up the following two schemas: {"name": "shortSchema", "namespace": "nifi", "type": "record", "fields": [ { "name": "a", "type": "string" }, { "name": "b", "type": "string" } ]} {"name": "fullSchema", "namespace": "nifi", "type": "record", "fields": [ { "name": "c", "type": ["null", "string"], "default" : "default value for field c" }, { "name": "d", "type": ["null", "string"], "default" : "default value for field d" }, { "name": "a", "type": "string" }, { "name": "b", "type": "string" } ]} Then made the following flow: What this flow does it the following: Generate a CSV with two rows and the columns a,b,c Reads the CSV using the shortSchema and writes as Avro with the shortSchema Updates the schema.name attribute to fullSchema Reads the Avro using the embedded schema (shortSchema) and writes it using the schema from schema.name (fullSchema) Reads the Avro using the embedded schema (fullSchema) and writes it back to CSV At the end the CSV that is printed now has the new fields with default values filled in. The CSV part is just for being able to easily see what is going on and is obviously not required if you already have Avro data. Here is a template of the flow: convert-avro-short-to-full.xml ... View more

The session provides methods to read and write to the flow file content. If you are reading only then session.read with an InputStreamCallback will give you an InputStream to the flow file content If you are writing only then session.write with an OutputStreamCallback will give you an OutputStream to the flow file content If you are reading and writing at the same time then a StreamCallback will give access to the both an InputStream and OutputStream In your case, if you are just looking to extract a value then you likely need an InputStreamCallback and you would use the InputStream to read the content and parse it appropriately for your data. You can look at examples in the existing processors: https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ExtractText.java#L313-L318 Keep in mind, the above example reads the whole content of the flow file into memory which can be dangerous when there are very large flow files, so whenever possible it is best to process the content in chunks. ... View more