Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

avatar
Contributor

Hi,

I have a three node HDF 2.0 unsecured cluster , and i would like to get some log file in another system.

I originally intend to test this :

SYSLOG ==> KAFKA ==> HDF ==> HDFS

But i have read this :

https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html

https://pierrevillard.com/tag/apache-minifi/

https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/bk_minifi-quick-start/content/index.html

And i was wondering maybe I can do this :

( MINIFI:HDF ) ==> KAFKA ==> HDF ==> HDFS

My question are:

IS MY USE CASE POSSIBLE WITH THE SECOND SCHEMAS ? AND IF IT IS HOW ?

I have tried the Getting started with MiNiFi in the first link but when i replace the config.yml with my HDF template ( transform with minifi toolkit and renamed config.yml ).

i get this from logs/minifi-app.log :

ERROR [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi

INFO [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise)

I'm missing many things here and need some help.

best regards,

Toky

1 ACCEPTED SOLUTION

avatar
Master Guru

I'm not really sure you need Kafka in this flow. I would consider:

MiNiFi -> NiFi -> HDFS

The way you connect MiNiFi to NiFi is through site-to-site using a Remote Process Group. See Joe Percivall's presentation here, slides 19 & 20:

http://www.slideshare.net/hortonworks/hortonworks-data-in-motion-webinar-series-part-6-edge-intellig...

Are there additional errors in your MiNiFi log?

It could be that one of the processors you used to create the flow in NiFi is not available in MiNiFi.

View solution in original post

5 REPLIES 5

avatar
Master Guru

I'm not really sure you need Kafka in this flow. I would consider:

MiNiFi -> NiFi -> HDFS

The way you connect MiNiFi to NiFi is through site-to-site using a Remote Process Group. See Joe Percivall's presentation here, slides 19 & 20:

http://www.slideshare.net/hortonworks/hortonworks-data-in-motion-webinar-series-part-6-edge-intellig...

Are there additional errors in your MiNiFi log?

It could be that one of the processors you used to create the flow in NiFi is not available in MiNiFi.

avatar
Master Guru

And this article that was just posted is a great example by @Ryan Cicak of setting up the communication between MiNiFi and NiFi https://community.hortonworks.com/articles/67756/ingesting-log-data-using-minifi-nifi.html

avatar
Contributor

Hi @Bryan Bende

Tanks for the answer, I'll look into those link

* this are my minifi extra log :

INFO [main] org.apache.nifi.minifi.MiNiFi Controller initialization took 798231885 nanoseconds. INFO [pool-1-thread-2] org.apache.nifi.BootstrapListener Received SHUTDOWN request from Bootstrap INFO [pool-1-thread-2] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... INFO [pool-1-thread-2] o.apache.nifi.controller.FlowController Initiated graceful shutdown of flow controller...waiting up to 10 seconds INFO [pool-1-thread-2] o.apache.nifi.controller.FlowController Controller has been terminated successfully. INFO [pool-1-thread-2] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap INFO [pool-1-thread-2] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise). INFO [Thread-1] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise).

* in my template i have Getfile ==> Updateattibute ==> RPG and those attribute are available in MINIFI.

* I've put KAFKA into the schemas just in case for bacjk up.

best regards,

Toky

avatar
Master Guru

When you say you "put Kafka into the schemas just in case" do you mean your MiNiFi flow has Kafka processors in it?

If so I would try removing that because I don't think MiNiFi has the Kafka processors available by default.

avatar
Contributor

No i would like to take some file log with (Minifi and nifi ) then writte the raw file to kafka