The flow you see in the UI comes from the conf directory flow.xml.gz, unrelated to the H2 database. If you installed two NiFi instances you should have two completely separate directories like /opt/nifi1 and /opt/nifi2, each with their own conf sub-directories, each with their own flow.xml.gz. If you had an existing NiFi instance with a flow, and then copied that directory to nifi2, it would start with the same flow, but from there if you made changes to either they should be separate. ... View more

You can attach the debugger from your IDE to NiFI... In NiF's conf directory in bootstrap.conf there is a line commented out like this: #java.arg.debug=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=8000 If you uncomment that line and restart NiFi, the Java process will be listening for a debug connection on port 8000. If you want the process to wait for a connection before starting, you can also set suspend=y. Connecting your IDE debugger to the Java process will be specific to which IDE you are using. You can also add addition logging statements in your processor code using the logger return from getLogger() to see how far it is getting. The log-levels are controlled through the logback.xml file in the conf directory. The default level for processors is WARN: <loggername="org.apache.nifi.processors"level="WARN"/> ... View more

I just realized something... GetKafka and PutKafka use the kafka-client 0.8.2, and there are newer processors in 0.7.0 called ConsumeKafka and PublishKafka which use kafka-client 0.9.0.1. Since you are using Kafka 0.9 I think we should be using ConsumeKafka here. Lets see if that works any better, and sorry for the confusion. ... View more

As you mentioned, NiFi does offer many capabilities that can be used to perform ETL functionality. In general though, NiFi is more of a general purpose dataflow tool. NiFi clusters usually scale to dozens of nodes, so if your "transform" needs to run on 100s (or 1000s) of nodes in parallel, then it may be better to use NiFi to bring data to another processing framework like Storm, Spark, etc. Similar for the "extract" part... NiFi has capabilities to extract from relational databases with ExecuteSQL and QueryDatabaseTable processors which can solve many uses case, and for more extreme use-cases something like sqoop can leverage a much larger Hadoop cluster to perform the extraction. So as always there is no single correct answer and it depends a lot on the use-case. ... View more