There are generally two things to consider. First, the configuration of a given dataflow. Second, the code required for a given dataflow. If your deployed artifacts in dev and prod are aligned then your focus is on configuration. For this NiFi supports templates which you can generate in dev and import into production. Templates are a great start for this but they do have the current downside in that they won't copy sensitive properties and they can be too coupled to environmental items like database URLs or web URLs. There are efforts underway in the Apache NiFi community to make them more portable via environment variable mappings which tie to a given environment and then the templates will tie to the mappings. In the case where you also need to get new deployment artifacts into production we benefit from Apache NiFi's support for easily deployed NiFi Archives (NARs) which nicely contain the code and dependencies so it is generally as easy as moving in a new NAR bundle into the lib. Typically people will have an 'extensions' folder and place their items in there. On restart NiFi will read its configuration classpath location(s) and make that new code available. In a cluster people typically add these items to all nodes then do rolling restart of the nodes in the cluster to avoid any downtime for the flow. ... View more

What looks likely to be happening is that your nar is bundling things you did not intend to include or should not include. Specifically you will not want to include the actual implementations of services you're depending on. You can have a nar depend on another nar and NiFi will automatically not actually pull that dependent nar into your nar but rather will wire it up at startup as part of its classpath handling. However, if you include the implementation of other services they could conflict as I suspect is the case here. Could you share/show details of how your project it structure or provide a pom and we could probably give you more specific pointers. There are several examples of nars in the apache nifi source too that show how to depend on services without pulling in their implementations. This is a good example https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-nar/pom.xml It pulls in the nar for the service definitions so that this nar can depend on those things but not actually include them in it's own package or create conflicts. ... View more

Hello Unfortunately this means the process NiFi was told to execute has not returned. In such a case there is an outstanding thread and we intentionally prevent additional instances from being started until this one is dealt with. From the stacktrace you provided we see "Timer-Driven Process Thread-9" Id=69 RUNNABLE (in native code) at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:272) at java.io.BufferedInputStream.read1(BufferedInputStream.java:273) at java.io.BufferedInputStream.read(BufferedInputStream.java:334) - waiting on java.lang.UNIXProcess$ProcessPipeInputStream@e6f0a2f This tells us we're sitting and waiting for that command to do something (finish, respond with data). It appears to be in a hung state. You'll need to restart NiFi and try to assess why that command isn't doing anything. What command are you trying to run? It might be that for your case the 'ExecuteProcess' processor is a better fit. Thanks Joe ... View more

Hello, Keep in mind each FlowFile is comprised of attributes and content. RouteOnAttribute is built specifically to look at FlowFile attributes. RouteOnContent looks at FlowFile content. In the case of JSON you can, for example, use EvaluateJSONPath to extract fields from the JSON to become FlowFile attributes. You can use SplitJSON to split the JSON bundles into invidual flowfiles then extract values and then route on attributes. The various processors offer you a variety of ways to route, transform, and deliver the data. Thanks Joe ... View more

This is due to this issue https://issues.apache.org/jira/browse/NIFI-990 which fixed the fact that the failure relationship was mistakenly not provided before. When importing templates made against the previous version they will not have that relationship checked since it wasn't there. So it will show as invalid until you check it as auto-terminate or use it. You may wish to recreate the template. ... View more

The HDF release does support interacting with Kerberized Kafka instances as found within the HDP stack. This is because HDP added support for Kerberized Kafka in advance of the community supporting it (Kafka 0.8.x). In Apache Kafka world now (0.9x) there is Kerberos support. So, - Apache NiFi supports non Kerberized Kafka clusters today. - HDF releases of NiFi have patched support for Kerberized Kafka clusters in HDP - Upcoming Apache NiFi releases will add support for the 0.9x Apache Kafka kerberos model Thanks Joe ... View more

I haven't tested this myself but NAS/SAN arrangements have worked quite well in the past. Needs testing to understand latencies/tradeoffs but frankly I suspect it will work just fine. ... View more

In the Get/PutKafka processors you should be able to add a dynamic property called 'fetch.message.max.bytes' and set the value you need. The processor should allow you to add dynamic properties which map to Kafka consumer properties and it will pass them to the consumer/producer config as needed. ... View more

Vance, We completely agree with you. NiFi already supports some powerful security and multi-role authorization capabilities. But as you mention we should support multiple different groups with different levels of access to various parts of the flow. That is an important roadmap item and work is underway. You can see a bit about the nifi community thinking on this wiki page https://cwiki.apache.org/confluence/display/NIFI/Multi-Tentant+Dataflow and there are related threads such as https://cwiki.apache.org/confluence/display/NIFI/Redesign+User+Interface and https://cwiki.apache.org/confluence/display/NIFI/Support+Authorizer+API If you need help setting up secure NiFi you can read more here https://community.hortonworks.com/articles/886/securing-nifi-step-by-step.html and in the administration guide https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#security-configuration Thanks Joe ... View more

You can pull data from JMS using either queues or topics by using these processors: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetJMSTopic/index.html https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetJMSQueue/index.html Those processors presently support ActiveMQ out of the box. It is pretty easy to add other vendors as well but we're going to make it even easier in an upcoming release. Once you pull data from JMS then you connect that processor's output to PutHDFS as explained here: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/index.html You might find these templates useful to help get you started: https://github.com/hortonworks-gallery/nifi-templates/tree/master/templates Thanks Joe ... View more

Hello You've run into a bug. https://issues.apache.org/jira/browse/NIFI-1396, https://issues.apache.org/jira/browse/NIFI-21 Has been around for quite a while but someone else ran into recently. Should be sorted soon but if you're interested in contributing a fix please let us know. Thanks Joe ... View more

Thanks for reporting it and for providing the stack traces. Very helpful. I've filed an Apache NiFi JIRA for it https://issues.apache.org/jira/browse/NIFI-1417 ... View more

Hello. In looking at the API/products facebook offers that URL api.facebook.com/restserver.php was not something I could find. Can you point to the Facebook API documentation of the particular endpoint you wish to access? Also, the programmableweb entry you provided shows that authentication in the form of oauth, apikey, username/password is required. But, it sounds like those weren't set. Might be better to just implement a process to listen to/interact with the Facebook API. Thanks Joe ... View more

Hello @Sunile Manjee. As Andrew mentions the http://nifi.apache.org/quickstart.html does outline how to alter the settings to ensure the NiFi process is able to have sufficient open files. One common gotcha is that the setting is tied to a different user than your NiFi process is executed as and in making sure that the session NiFi starts with does indeed have that new open files setting reflected. Are you running 'bin/nifi.sh start' in the same terminal that you run 'ulimit -a' to see if the setting has taken effect? Another good technique you can use is to run 'lsof -p 12345' assuming the pid of NiFi is 12345 and it will show you all the open file handles that NiFi process has. Thanks Joe ... View more

Absolutely. Don't have a good timeline to offer at the moment but it is on the roadmap. Happy to talk it over further with you. Thanks! ... View more

Hello What the NiFi server exposes/provides is a RESTFul API. If you attach your web browser to it you'll get a visually friendly UI. You can lock down access via 2-Way SSL in HDF 1.0 and in the upcoming HDF 1.1 you can also do username/password with active directory or directory server via LDAP. Thanks Joe ... View more