Interesting, can you confirm which version of NiFi? I'm assuming 1.1 since you have the new UI with colors. ... View more

There are currently a couple of options... If you just want to figure out where a flow file came from for troubleshooting/debugging, then using provenance can tell you this by looking at the transit URI of the RECEIVE event. If you want to make a decision somewhere in your NiFi flow based on which MiNiFi sent the flow file, then currently you would need set an attribute on the MiNiFi side like "minifi.host = ${hostname()}" in an UpdateAttribute processor, so that when it got transferred to NiFi that attribute would be there. There is a pull request open to make NiFi automatically create this attribute for you when receiving the flow files via site-to-site, so basically the same info that is available on the RECEIVE event would be available in attributes: https://issues.apache.org/jira/browse/NIFI-2585 https://github.com/apache/nifi/pull/1320 ... View more

To go from CSV or JSON to Avro you would use ConvertJsonToAvro or ConvertCsvToAvro, there is no conversion for XML currently. With ConvertJsonToAvro and ConvertCsvToAvro you can directly enter the schema in the properties of the processor, or you can attempt to use InferAvroSchema before them. InferAvroSchema will guess a schema and put it into a flow file attribute which you can then reference as ${inferred.avro.schema} in the conversion processor. For ConvertAvroSchema you can either enter schemas directly into the properties, or you can reference flow file attributes using expression language (if you have the schemas in flow file attributes). ... View more

NONE means don't use a codec at all. AUTOMATIC is not meant to be used with PutHDFS, it is only for the read side (GetHDFS and FetchHDFS) DEFAULT means use the DefaultCodec class provided by Hadoop which I believe uses a zlib compression (slide 6 here http://www.slideshare.net/Hadoop_Summit/kamat-singh-june27425pmroom210cv2) ... View more

The data in the content repository should be controlled through the settings in nifi.properties: # Content Repository nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository nifi.content.claim.max.appendable.size=10 MB nifi.content.claim.max.flow.files=100 nifi.content.repository.directory.default=./content_repository nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true nifi.content.repository.always.sync=false nifi.content.viewer.url=/nifi-content-viewer/ Is the data in the archive folders (underneath each of the folders in the content_repository)? If so you can turn off archiving using the property above, or reduce the thresholds for how long to hold on to archived data. If the data is not in the archive folders then it is still deemed to be active in the flow, since multiple flowfiles are bundled together in a content claim, you can have a content claim with some flow files still active and it can't archive the whole thing until they are all no longer in the flow. ... View more

Yes that is a lot less than 700 devices generating 1GB each per hour 🙂 Most people want some kind of redundancy for production so you may still want to a smaller cluster, say 3 nodes, but yes you should be able to do that on one node. ... View more