question Re: how to read my hdfs data into nifi cluster in Support Questions

how to read my hdfs data into nifi cluster

Former Member — Sat, 25 Jun 2016 18:25:16 GMT

i created two different clusters

1) two node nifi cluster 2) three node hadoop (HDP) cluster and i want to load my hdfs data into nifi cluster. so please explain what are the configurations we need to change and how to connect nifi and hadoop cluster.

Re: how to read my hdfs data into nifi cluster

jgeorge — Sun, 18 Aug 2019 12:15:18 GMT

Hi Kishore,

You could drop below processors as per your requirement on to your NiFi UI and configure them properly to pull the data from HDFS.

and to configure them, add hdfs, core property files location saved on to the local file system on nifi node(s), can use Ambari to download these from HDP cluster. pls find sample conf screenshot below:

Thanks!

Re: how to read my hdfs data into nifi cluster

Former Member — Mon, 27 Jun 2016 23:53:55 GMT

Hi jobin,

1) i installed NiFi outside Hadoop. so core-site.xml and hdfs-site.xml files i want to put in nifi bin folder or some other folder.and which directory i want to add in cofgure processor (like /nifi/bin/core-site.xml or /etc/hadoop/conf/core-site.xml).

2) How does it work when you have data to be streamed ? Should it go to the first NiFi node ? If I have 100 MB/s data how we share a load between these two systems ? Do you need a load balancer ?

please explain how to resolve this issues .

Re: how to read my hdfs data into nifi cluster

TimothySpann — Tue, 28 Jun 2016 00:29:31 GMT

split your ingest into different NIFI nodes depending on their location. from there do some initial clean up, send it to kafka which can send to your remote hadoop cluster for landing in HDFS.

Re: how to read my hdfs data into nifi cluster

Former Member — Tue, 28 Jun 2016 01:17:06 GMT

hi timothy ,

my nifi cluster is different and hadoop cluster is different so how to connect these two cluster and where to save xml file in nifi.

Re: how to read my hdfs data into nifi cluster

jgeorge — Tue, 28 Jun 2016 01:27:09 GMT

Hi @kishore sanchina,

You can save config files in any directory on nifi node, and provide that path in the processor configuration.

Re: how to read my hdfs data into nifi cluster

Former Member — Tue, 28 Jun 2016 02:05:52 GMT

thanks jobin 🙂

Re: how to read my hdfs data into nifi cluster

Former Member — Tue, 28 Jun 2016 02:08:51 GMT

How does it work when you have data to be streamed ? Should it go to the first NiFi node ? If I have 100 MB/s data how we share a load between these two systems ? Do you need a load balancer ?

Re: how to read my hdfs data into nifi cluster

Former Member — Tue, 28 Jun 2016 02:08:56 GMT

How does it work when you have data to be streamed ? Should it go to the first NiFi node ? If I have 100 MB/s data how we share a load between these two systems ? Do you need a load balancer ?

Re: how to read my hdfs data into nifi cluster

jgeorge — Tue, 28 Jun 2016 12:51:39 GMT

Hi Kishore,

If its a cluster, You will be creating your flows in NCM[Nifi Cluster Manager] UI, which runs on all the nodes in the cluster. Since you have only 2 nodes in the cluster(may be only one worker node and NCM), you may not have much to load balance there. Still you can stimulate a load balancer with nifi site-to- site protocol.

you can get more info on site-to-site protocol load balancing here:

https://community.hortonworks.com/questions/509/site-to-site-protocol-load-balancing.html

Thanks!