This is an extremely rough guess and it really depends on a lot of factors, but if you assume all devices produce 1GB an hour thats 700GB/hour, thats about 200MBs/second. You are probably looking at a 7-8 node NiFi cluster, where each node has 16-24 cpus, 16+ GB RAM, 10 Gigabit NICs. You will also need to ensure proper separation of the repositories on different disks. ... View more

Unfortunately not every property currently supports using the variable registry. The way you can tell is in the documentation for a processor, by looking at the property description. If it says "Supports Expression Language: true" then you can reference variables. For example, with PutHDFS it looks like only the Directory property currently supports it: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/index.html ... View more

Since NiFi 1.0, there are two types of controller services... You can create controller services from the global hamburger menu in the top right, but these are only to be used by any reporting tasks that use a controller service. For example, the site to site provenance reporting task can use an SSL context service. For processors they have to be created through the context pallet on the left hand side. If you are on the root canvas and have nothing selected, and then click the configuration icon in the pallet then you can create a controller service at the root group. These should be visible to any processors on the root group and with in any sub groups. If you go into a process group and create a controller service there, then it is visible to anything in that group and below, but not anything above it. ... View more

Each CSV or JSON that comes in the InferAvroSchema could be different so it will infer the schema for each flow file and put the schema where you specify the schema destination, either flow file content or a flow file attribute. Then you can use that attribute in ConvertCsvToAvro as the schema by referencing ${inferred.avro.schema}. If you are sending only one type of CSV in to ConvertCsvToAvro then it would be more efficient for you to define the Avro schema you want and not use InferAvroSchema. ... View more

This is a known problem with PutKafka: https://issues.apache.org/jira/browse/NIFI-2671 What versions of Kafka and NiFi are you using? PutKafka should only be used with a 0.8 Kafka broker, PublishKafka and PublishKafka_0_10 should be used with Kafka 0.9 and 0.10 respectively. For PublishKafka and PublishKafka_0_10, they were fixed to send messages larger than 1MB: https://issues.apache.org/jira/browse/NIFI-2614 ... View more