question Re: MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ? in Archives of Support Questions (Read Only)

MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

raobelina_toky_ — Tue, 22 Nov 2016 18:01:43 GMT

Hi,

I have a three node HDF 2.0 unsecured cluster , and i would like to get some log file in another system.

I originally intend to test this :

SYSLOG ==> KAFKA ==> HDF ==> HDFS

But i have read this :

https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html

https://pierrevillard.com/tag/apache-minifi/

https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/bk_minifi-quick-start/content/index.html

And i was wondering maybe I can do this :

( MINIFI:HDF ) ==> KAFKA ==> HDF ==> HDFS

My question are:

IS MY USE CASE POSSIBLE WITH THE SECOND SCHEMAS ? AND IF IT IS HOW ?

I have tried the Getting started with MiNiFi in the first link but when i replace the config.yml with my HDF template ( transform with minifi toolkit and renamed config.yml ).

i get this from logs/minifi-app.log :

ERROR [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi

INFO [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise)

I'm missing many things here and need some help.

best regards,

Toky

Re: MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

bbende — Tue, 22 Nov 2016 23:24:43 GMT

I'm not really sure you need Kafka in this flow. I would consider:

MiNiFi -> NiFi -> HDFS

The way you connect MiNiFi to NiFi is through site-to-site using a Remote Process Group. See Joe Percivall's presentation here, slides 19 & 20:

http://www.slideshare.net/hortonworks/hortonworks-data-in-motion-webinar-series-part-6-edge-intelligence-iot-minifi?ref=http://hortonworks.com/blog/edge-intelligence-iot-apache-minifi/

Are there additional errors in your MiNiFi log?

It could be that one of the processors you used to create the flow in NiFi is not available in MiNiFi.

Re: MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

bbende — Tue, 22 Nov 2016 23:26:24 GMT

And this article that was just posted is a great example by @Ryan Cicak of setting up the communication between MiNiFi and NiFi https://community.hortonworks.com/articles/67756/ingesting-log-data-using-minifi-nifi.html

Re: MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

raobelina_toky_ — Tue, 22 Nov 2016 23:59:41 GMT

Hi @Bryan Bende

Tanks for the answer, I'll look into those link

* this are my minifi extra log :

INFO [main] org.apache.nifi.minifi.MiNiFi Controller initialization took 798231885 nanoseconds. INFO [pool-1-thread-2] org.apache.nifi.BootstrapListener Received SHUTDOWN request from Bootstrap INFO [pool-1-thread-2] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... INFO [pool-1-thread-2] o.apache.nifi.controller.FlowController Initiated graceful shutdown of flow controller...waiting up to 10 seconds INFO [pool-1-thread-2] o.apache.nifi.controller.FlowController Controller has been terminated successfully. INFO [pool-1-thread-2] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap INFO [pool-1-thread-2] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise). INFO [Thread-1] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise).

* in my template i have Getfile ==> Updateattibute ==> RPG and those attribute are available in MINIFI.

* I've put KAFKA into the schemas just in case for bacjk up.

best regards,

Toky

Re: MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

bbende — Wed, 23 Nov 2016 00:16:19 GMT

When you say you "put Kafka into the schemas just in case" do you mean your MiNiFi flow has Kafka processors in it?

If so I would try removing that because I don't think MiNiFi has the Kafka processors available by default.

Re: MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

raobelina_toky_ — Wed, 23 Nov 2016 18:26:11 GMT

No i would like to take some file log with (Minifi and nifi ) then writte the raw file to kafka