In general, concurrent tasks is the number of threads calling onTrigger for an instance of a processors. In a cluster, if you set concurrent tasks to 4, then it is 4 threads on each node of your cluster. I am not as familiar with all the ins and outs of the kafka processors, but for GetKafka it does something like this: int concurrentTaskToUse = context.getMaxConcurrentTasks(); final Map<String, Integer> topicCountMap = new HashMap<>(1); topicCountMap.put(topic, concurrentTaskToUse); final Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap); The consumer is from the Kafka 0.8 client, so it is creating a message-stream for each concurrent task. Then when the processor is triggered it takes one of those message-streams and consumes a message, and since multiple concurrent tasks are trigger the processor, it is consuming from each of those streams in parallel. As far as rebalancing, I think the Kafka client does that transparently to NiFi, but I am not totally sure. Messages that have already been pulled into a NiFi node will stay there until the node is back up and processing. ... View more

I think it depends what you mean by "schedule it to run every hour"... NiFi itself would always be running and different processors can be scheduled to run according to their needs. Every processor supports timer based scheduling or cron based scheduling, so using either of those you can set a source processor to run every hour. You could also use the REST API to start and stop processors as needed, anything you can do in the UI can be done through the REST API. For best practices for upgrading NiFi see this wiki page: https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi Deploying changes to production, there are a couple of approaches, one of them is based around templates: https://github.com/aperepel/nifi-api-deploy Some people also just move the flow.xml.gz from one environment to another, but this assumes you have parametized everything that is different between environments. ... View more

The Input Directory property of GetFile supports Expression Language so you can reference a system property like ${my.directory} and define my.directory in bootstrap.conf by adding another java arg like: java.arg.15=-Dmy.directory=/foo Then you can have different bootstrap files per environment. ... View more

The ListFile processor keeps track of files it has already seen and only picks up files where the modified date is newer than the last time the processor ran. ListFile produces a flow file for each path to fetch and is used with FetchFile to actually retrieve the file. ... View more

You can provide additional dependencies to the ExecuteScript processor by using the "Module Directory" property as described here: http://funnifi.blogspot.com/2016/02/executescript-using-modules.html You generally shouldn't put any jars into NiFi's lib directory because that can impact all other NARs. ... View more