question how can I send send JMS to HDFS using nifi?? in Archives of Support Questions (Read Only)

how can I send send JMS to HDFS using nifi??

pansheriya_vija — Fri, 22 Jan 2016 02:59:52 GMT

Re: how can I send send JMS to HDFS using nifi??

aervits — Fri, 22 Jan 2016 03:02:19 GMT

you need a processor to read from JMS queue and you need a processor to put data into hdfs. you may need to handle your logic a bit but that's basically the processors you need @vijay patel

Re: how can I send send JMS to HDFS using nifi??

JoeWitt — Fri, 22 Jan 2016 03:06:00 GMT

You can pull data from JMS using either queues or topics by using these processors:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetJMSTopic/index.html

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetJMSQueue/index.html

Those processors presently support ActiveMQ out of the box. It is pretty easy to add other vendors as well but we're going to make it even easier in an upcoming release.

Once you pull data from JMS then you connect that processor's output to PutHDFS as explained here:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/index.html

You might find these templates useful to help get you started:

https://github.com/hortonworks-gallery/nifi-templates/tree/master/templates

Thanks

Joe

Re: how can I send send JMS to HDFS using nifi??

pansheriya_vija — Fri, 22 Jan 2016 03:19:40 GMT

for download nifi I can do it on windows or linux? thanks for answer

Re: how can I send send JMS to HDFS using nifi??

aervits — Fri, 22 Jan 2016 03:24:05 GMT

@vijay patel you can run nifi on anything that can run jvm.

Re: how can I send send JMS to HDFS using nifi??

mpayne — Fri, 22 Jan 2016 03:32:41 GMT

You may also want to consider that JMS messages tend to be quite small, and before pushing to HDFS it may help to merge together many JMS messages so that you don't have a huge number of files in HDFS. You could use the MergeContent Processor to accomplish this.

Re: how can I send send JMS to HDFS using nifi??

pansheriya_vija — Fri, 22 Jan 2016 06:16:32 GMT

Thanks mpayne

Re: how can I send send JMS to HDFS using nifi??

vvaks — Mon, 28 Mar 2016 06:56:25 GMT

Configure ConsumeJMS processor, configure MergeContent processor to create suitable file sizes for HDFS, the use PutHDFS processor to write to HDFS directly.