Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

Solved Go to solution
Highlighted

MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

Contributor

Hi,

I have a three node HDF 2.0 unsecured cluster , and i would like to get some log file in another system.

I originally intend to test this :

SYSLOG ==> KAFKA ==> HDF ==> HDFS

But i have read this :

https://community.hortonworks.com/articles/56341/getting-started-with-minifi.html

https://pierrevillard.com/tag/apache-minifi/

https://docs.hortonworks.com/HDPDocuments/HDF2/HDF-2.0.1/bk_minifi-quick-start/content/index.html

And i was wondering maybe I can do this :

( MINIFI:HDF ) ==> KAFKA ==> HDF ==> HDFS

My question are:

IS MY USE CASE POSSIBLE WITH THE SECOND SCHEMAS ? AND IF IT IS HOW ?

I have tried the Getting started with MiNiFi in the first link but when i replace the config.yml with my HDF template ( transform with minifi toolkit and renamed config.yml ).

i get this from logs/minifi-app.log :

ERROR [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi

INFO [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise)

I'm missing many things here and need some help.

best regards,

Toky

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

I'm not really sure you need Kafka in this flow. I would consider:

MiNiFi -> NiFi -> HDFS

The way you connect MiNiFi to NiFi is through site-to-site using a Remote Process Group. See Joe Percivall's presentation here, slides 19 & 20:

http://www.slideshare.net/hortonworks/hortonworks-data-in-motion-webinar-series-part-6-edge-intellig...

Are there additional errors in your MiNiFi log?

It could be that one of the processors you used to create the flow in NiFi is not available in MiNiFi.

View solution in original post

5 REPLIES 5
Highlighted

Re: MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

I'm not really sure you need Kafka in this flow. I would consider:

MiNiFi -> NiFi -> HDFS

The way you connect MiNiFi to NiFi is through site-to-site using a Remote Process Group. See Joe Percivall's presentation here, slides 19 & 20:

http://www.slideshare.net/hortonworks/hortonworks-data-in-motion-webinar-series-part-6-edge-intellig...

Are there additional errors in your MiNiFi log?

It could be that one of the processors you used to create the flow in NiFi is not available in MiNiFi.

View solution in original post

Highlighted

Re: MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

And this article that was just posted is a great example by @Ryan Cicak of setting up the communication between MiNiFi and NiFi https://community.hortonworks.com/articles/67756/ingesting-log-data-using-minifi-nifi.html

Highlighted

Re: MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

Contributor

Hi @Bryan Bende

Tanks for the answer, I'll look into those link

* this are my minifi extra log :

INFO [main] org.apache.nifi.minifi.MiNiFi Controller initialization took 798231885 nanoseconds. INFO [pool-1-thread-2] org.apache.nifi.BootstrapListener Received SHUTDOWN request from Bootstrap INFO [pool-1-thread-2] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... INFO [pool-1-thread-2] o.apache.nifi.controller.FlowController Initiated graceful shutdown of flow controller...waiting up to 10 seconds INFO [pool-1-thread-2] o.apache.nifi.controller.FlowController Controller has been terminated successfully. INFO [pool-1-thread-2] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap INFO [pool-1-thread-2] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise). INFO [Thread-1] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server... [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise).

* in my template i have Getfile ==> Updateattibute ==> RPG and those attribute are available in MINIFI.

* I've put KAFKA into the schemas just in case for bacjk up.

best regards,

Toky

Highlighted

Re: MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

When you say you "put Kafka into the schemas just in case" do you mean your MiNiFi flow has Kafka processors in it?

If so I would try removing that because I don't think MiNiFi has the Kafka processors available by default.

Re: MINIFI AND HDF 2.0 CONNECTING TOGETHER TO GET FILE LOCALLY ?

Contributor

No i would like to take some file log with (Minifi and nifi ) then writte the raw file to kafka

Don't have an account?
Coming from Hortonworks? Activate your account here