@rich @William Gonzalez any updates on the Certified Professional Data Engineer (HCPDE) ? ... View more

Makes sense, I think haproxy (http://www.haproxy.org/) is a free load balancer that supports TCP, then your data producer can just send to the haproxy address. ... View more

Try adding this to pom.xml of nar folder of your project <groupId>org.apache.nifi</groupId> <artifactId>nifi-standard-nar</artifactId> <version>1.3.0</version> <type>nar</type> , Try adding <dependency> <groupId>org.apache.nifi</groupId> <artifactId>nifi-standard-nar</artifactId> <version>1.3.0</version> <type>nar</type> </dependency> to pom.xml of nar folder of NiFi project ... View more

Here is the link to a short tutorial with Waterline using a HDP Sandbox created jointly by Hortonworks and Waterline: Manage your Data Lake more efficiently with Waterline Data Inventory and HDP ... View more

In case of a secure NiFi instance, you can check for heartbeat with this curl command below curl 'https://<nifi-server>:8077/nifi-api/access/token' -H 'Content-Type: application/x-www-form-urlencoded; charset=UTF-8' --data 'username=<username>&password=<password>' --compressed --insecure if you get a token back, the NiFi instance is up and running. Here is the link where this is posted/answered - https://community.hortonworks.com/questions/96713/nifi-api-unable-to-validate-the-access-token-error.html ... View more

Thanks @Matt Clarke ... View more

While there are no official NiFi tasks (processor, controller services, reporting tasks, etc.) users can integrate NiFi and Atlas for data lineage using processors that come out of the box with NiFi. UpdateAttribute, ReplaceTExt, InokeHTTP, EvaluateJsonPath can be stitched together to invoke the Atlas API to populate lineage information. Wrapping this up in a template/progress group, one can productionise the flow. That said, Vadim's solution is a much more elegant implementation that can be extended. I'm sure this will be the approach taken by the NiFi development team in their implementation of the NiFi-Atlas bridge. ... View more

Thank you @Pierre Villard I think I'm getting there... 🙂 ... View more

thank you @kkawamura for the heads up on the new processor. ... View more

Hi Kylo is providing the functionality you are looking for They have a concept together with nifi which is called reusable flow This flow can be connected to any feed templates and in the end all feeds hook into the same reusable flow in nifi So in short you create a flow in nifi with input port, create a template of this flow the reusable flow, export it and inport the file as a reusable flow in kylo. Create your feed template flows in nifi and create template of them, import the template from nifi in kylo and connect the output port of your template to the input port of the reusable flow. Create a feed in kylo from your template. Kilo will then generate a feed based on your feed template in nifi and connect this to the reusable flow. You can now modify your reusable flow and create a new template in nifi with the same name as the previous version of the reusable flow, export it to a file and import it to kylo and all your feeds are now connected to the new version of your reusable flow. If you do not want to use Kylo there is an alternative you create a group called reusable flow In this group you create a group called reusable flow v1 You connect your feeds to the input port on the reusable flow group And inside of the reusable flow you connect it to the v1 groups input port Now you want to make a new version of your reusable flow You make a copy of the group v1 to v2 Make the changes you want to do inside of v2, when you are satisfied you disconnect v1 and connect v2 instead. In this way it is very easy to switch to a new version without having to change any of your feeds. Below is link to youtube that describe the kylo reusable flow, but if you are only in nifi I recommend the groups in groups to maintain different versions as you then can easily switch back and forward without impacting all your feeds https://www.youtube.com/watch?v=Vj641MRJCd8 ... View more