Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Can we integrate NiFi with other Apache Software?

Solved Go to solution
Highlighted

Can we integrate NiFi with other Apache Software?

New Contributor
 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: Can we integrate NiFi with other Apache Software?

Rising Star

@Sunil Mukati

Your question is very broad, so if you need more info please be specific.

Based on the question tags I'm going to assume you're asking about NiFi integration with Spark, Kafka, Storm, and Solr.

The short answer is yes - we can integrate NiFi with other Apache Software :)

NiFi provides an easy way to stream data between different systems, and has in-built processors for dealing with most of the common Apache stack.

Kafka

NiFi has in-built processors to stream into and read data from Kafka:

  • PutKafka
  • GetKafka
  • PublishKafka
  • ConsumeKafka

I'd suggest checking out the following tutorial to get you started:

http://hortonworks.com/hadoop-tutorial/realtime-event-processing-nifi-kafka-storm/

Solr

NiFi has a PutSolrContentStream processor that allows you to stream data directly into a Solr index. Check out the following tutorial that uses NiFi to index twitter data directly into Solr and visualises it using banana:

https://community.hortonworks.com/articles/1282/sample-hdfnifi-flow-to-push-tweets-into-solrbanana.h...

Spark

Spark doesn't supply a mechanism to have data pushed to it, and instead pulls from other sources. You can integrate NiFi directly with Spark Streaming by exposing an Output Port in NiFi that Spark can consume from. The following article explains how to set up this integration:

https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html

Note that typical streaming architecture involves NiFi pushing data to Kafka, and Spark Streaming (or Storm) reading from Kafka.

Storm

As above, typically NiFi is integrated with Storm with Kafka acting as the message buffer. The tutorial realtime event processing tutorial linked above covers the details of building a streaming application in NiFi Kafka & Storm.

2 REPLIES 2

Re: Can we integrate NiFi with other Apache Software?

Hi,

Yes NiFi, by default, offers a very large number of processors to interact with other Apache softwares. Here is a non exhaustive list: Spark, Storm, Hive, HBase, Cassandra, Solr, Ignite, HDFS, Kafka, etc. You have the full list of available processors here:

https://nifi.apache.org/docs/nifi-docs/

Re: Can we integrate NiFi with other Apache Software?

Rising Star

@Sunil Mukati

Your question is very broad, so if you need more info please be specific.

Based on the question tags I'm going to assume you're asking about NiFi integration with Spark, Kafka, Storm, and Solr.

The short answer is yes - we can integrate NiFi with other Apache Software :)

NiFi provides an easy way to stream data between different systems, and has in-built processors for dealing with most of the common Apache stack.

Kafka

NiFi has in-built processors to stream into and read data from Kafka:

  • PutKafka
  • GetKafka
  • PublishKafka
  • ConsumeKafka

I'd suggest checking out the following tutorial to get you started:

http://hortonworks.com/hadoop-tutorial/realtime-event-processing-nifi-kafka-storm/

Solr

NiFi has a PutSolrContentStream processor that allows you to stream data directly into a Solr index. Check out the following tutorial that uses NiFi to index twitter data directly into Solr and visualises it using banana:

https://community.hortonworks.com/articles/1282/sample-hdfnifi-flow-to-push-tweets-into-solrbanana.h...

Spark

Spark doesn't supply a mechanism to have data pushed to it, and instead pulls from other sources. You can integrate NiFi directly with Spark Streaming by exposing an Output Port in NiFi that Spark can consume from. The following article explains how to set up this integration:

https://community.hortonworks.com/articles/12708/nifi-feeding-data-to-spark-streaming.html

Note that typical streaming architecture involves NiFi pushing data to Kafka, and Spark Streaming (or Storm) reading from Kafka.

Storm

As above, typically NiFi is integrated with Storm with Kafka acting as the message buffer. The tutorial realtime event processing tutorial linked above covers the details of building a streaming application in NiFi Kafka & Storm.

Don't have an account?
Coming from Hortonworks? Activate your account here