MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

[#1]

Hi,

I have a three node HDF 2.0 unsecured cluster , and i would like to get some log file in another system.

I originally intend to test this :

SYSLOG ==> KAFKA ==> HDF ==> HDFS

But i have read this :

https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html

https://pierrevillard.com/tag/apache-minifi/

https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/bk_minifi-quick-start/content/index.html

And i was wondering maybe I can do this :

( MINIFI:HDF ) ==> KAFKA ==> HDF ==> HDFS

My question are:

IS MY USE CASE POSSIBLE WITH THE SECOND SCHEMAS ? AND IF IT IS HOW ?

I have tried the Getting started with MiNiFi in the first link but when i replace the config.yml with my HDF template ( transform with minifi toolkit and renamed config.yml ).

i get this from logs/minifi-app.log :

ERROR [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi

INFO [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise)

I'm missing many things here and need some help.

best regards,

Toky

[#2 · Accepted solution]

I'm not really sure you need Kafka in this flow. I would consider:

MiNiFi -> NiFi -> HDFS

The way you connect MiNiFi to NiFi is through site-to-site using a Remote Process Group. See Joe Percivall's presentation here, slides 19 & 20:

http://www.slideshare.net/hortonworks/hortonworks-data-in-motion-webinar-series-part-6-edge-intellig...

Are there additional errors in your MiNiFi log?

It could be that one of the processors you used to create the flow in NiFi is not available in MiNiFi.

[#3]

And this article that was just posted is a great example by @Ryan Cicak of setting up the communication between MiNiFi and NiFi https://community.hortonworks.com/articles/67756/ingesting-log-data-using-minifi-nifi.html

[#4]

Hi @Bryan Bende

Tanks for the answer, I'll look into those link

* this are my minifi extra log :

INFO [main] org.apache.nifi.minifi.MiNiFi Controller initialization took 798231885 nanoseconds. INFO [pool-1-thread-2] org.apache.nifi.BootstrapListener Received SHUTDOWN request from Bootstrap INFO [pool-1-thread-2] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... INFO [pool-1-thread-2] o.apache.nifi.controller.FlowController Initiated graceful shutdown of flow controller...waiting up to 10 seconds INFO [pool-1-thread-2] o.apache.nifi.controller.FlowController Controller has been terminated successfully. INFO [pool-1-thread-2] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap INFO [pool-1-thread-2] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise). INFO [Thread-1] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise).

* in my template i have Getfile ==> Updateattibute ==> RPG and those attribute are available in MINIFI.

* I've put KAFKA into the schemas just in case for bacjk up.

best regards,

Toky

[#5]

When you say you "put Kafka into the schemas just in case" do you mean your MiNiFi flow has Kafka processors in it?

If so I would try removing that because I don't think MiNiFi has the Kafka processors available by default.

[#6]

No i would like to take some file log with (Minifi and nifi ) then writte the raw file to kafka