Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi for RabbitMQ and Couchbase

Solved Go to solution
Highlighted

Nifi for RabbitMQ and Couchbase

New Contributor

Can someone please confirm if I can use a NiFi to load data into HDP 2.3 from

1.MS SQL (To Hive if possible). I know I can Sqoop it, but having another option to load data using NiFi would add more value to introduce NiFi.

2.CouchDB

3.RabbitMQ

Thanks

Sundar R

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Nifi for RabbitMQ and Couchbase

Guru
@sraghavan@hortonworks.com

You can handle 1 and 2 out of the box with NiFi using the steps listed below. RabbitMQ is not currently supported natively.

1) MS SQL - Create DBCPConnectionPool in NiFi by going to "Controller Settings" -> "+" -> "DBCPConnectionPool". Here you can define your connection string and point to the SQLServer JDBC jar. After you define the connection pool return to your flow and insert a "ExecuteSQL" Processor that references the DBCPConnectionPool you just created. From there it is pretty straightforward. Keep in mind the output from "ExectueSQL" is Avro so you might want to use a "ConvertAvroToJSON" Processor if you don't want to use Avro. At this point you would use a "PutHDFS" to place the data in the Hive appropriate directory in HDFS.

2) CouchDB - Exact same process as MS SQL except you would use the JDBC driver for CouchDB when setting up the DBCPConnectionPool.

3) NiFi (as of version 0.3.0) only supports ActiveMQ out of the box. Creating your own custom Processor in Java to use in NiFi is supported however. You could create a RabbitMQ Processor to use if this is really needed.

3 REPLIES 3

Re: Nifi for RabbitMQ and Couchbase

Cloudera Employee

@sraghavan@hortonworks.com

I think the point of tags should be to separate the various components/keywords. Your tag 'nifi for rabitmq and couchbase' isn't really a single tag. I have re-tagged with 3 separate tags as separate words: Nifi rabbitmq and couchbase. Thanks.

Re: Nifi for RabbitMQ and Couchbase

Guru
@sraghavan@hortonworks.com

You can handle 1 and 2 out of the box with NiFi using the steps listed below. RabbitMQ is not currently supported natively.

1) MS SQL - Create DBCPConnectionPool in NiFi by going to "Controller Settings" -> "+" -> "DBCPConnectionPool". Here you can define your connection string and point to the SQLServer JDBC jar. After you define the connection pool return to your flow and insert a "ExecuteSQL" Processor that references the DBCPConnectionPool you just created. From there it is pretty straightforward. Keep in mind the output from "ExectueSQL" is Avro so you might want to use a "ConvertAvroToJSON" Processor if you don't want to use Avro. At this point you would use a "PutHDFS" to place the data in the Hive appropriate directory in HDFS.

2) CouchDB - Exact same process as MS SQL except you would use the JDBC driver for CouchDB when setting up the DBCPConnectionPool.

3) NiFi (as of version 0.3.0) only supports ActiveMQ out of the box. Creating your own custom Processor in Java to use in NiFi is supported however. You could create a RabbitMQ Processor to use if this is really needed.

Re: Nifi for RabbitMQ and Couchbase

Guru

There is now a processor for Couchbase and RabbitMQ is an AMQP messaging platform for which there is also a processor.