Nifi for RabbitMQ and Couchbase

[#1]

Can someone please confirm if I can use a NiFi to load data into HDP 2.3 from

1.MS SQL (To Hive if possible). I know I can Sqoop it, but having another option to load data using NiFi would add more value to introduce NiFi.

2.CouchDB

3.RabbitMQ

Thanks

Sundar R

[#2]

@sraghavan@hortonworks.com

I think the point of tags should be to separate the various components/keywords. Your tag 'nifi for rabitmq and couchbase' isn't really a single tag. I have re-tagged with 3 separate tags as separate words: Nifi rabbitmq and couchbase. Thanks.

[#3 · Accepted solution]

@sraghavan@hortonworks.com

You can handle 1 and 2 out of the box with NiFi using the steps listed below. RabbitMQ is not currently supported natively.

1) MS SQL - Create DBCPConnectionPool in NiFi by going to "Controller Settings" -> "+" -> "DBCPConnectionPool". Here you can define your connection string and point to the SQLServer JDBC jar. After you define the connection pool return to your flow and insert a "ExecuteSQL" Processor that references the DBCPConnectionPool you just created. From there it is pretty straightforward. Keep in mind the output from "ExectueSQL" is Avro so you might want to use a "ConvertAvroToJSON" Processor if you don't want to use Avro. At this point you would use a "PutHDFS" to place the data in the Hive appropriate directory in HDFS.

2) CouchDB - Exact same process as MS SQL except you would use the JDBC driver for CouchDB when setting up the DBCPConnectionPool.

3) NiFi (as of version 0.3.0) only supports ActiveMQ out of the box. Creating your own custom Processor in Java to use in NiFi is supported however. You could create a RabbitMQ Processor to use if this is really needed.

[#4]

There is now a processor for Couchbase and RabbitMQ is an AMQP messaging platform for which there is also a processor.