Support Questions
Find answers, ask questions, and share your expertise

how can I send send JMS to HDFS using nifi??

 
1 ACCEPTED SOLUTION

Accepted Solutions

Re: how can I send send JMS to HDFS using nifi??

You can pull data from JMS using either queues or topics by using these processors:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetJMSTopic/in...

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetJMSQueue/in...

Those processors presently support ActiveMQ out of the box. It is pretty easy to add other vendors as well but we're going to make it even easier in an upcoming release.

Once you pull data from JMS then you connect that processor's output to PutHDFS as explained here:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/index.ht...

You might find these templates useful to help get you started:

https://github.com/hortonworks-gallery/nifi-templates/tree/master/templates

Thanks

Joe

View solution in original post

7 REPLIES 7

Re: how can I send send JMS to HDFS using nifi??

Mentor

you need a processor to read from JMS queue and you need a processor to put data into hdfs. you may need to handle your logic a bit but that's basically the processors you need @vijay patel

Re: how can I send send JMS to HDFS using nifi??

You can pull data from JMS using either queues or topics by using these processors:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetJMSTopic/in...

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.GetJMSQueue/in...

Those processors presently support ActiveMQ out of the box. It is pretty easy to add other vendors as well but we're going to make it even easier in an upcoming release.

Once you pull data from JMS then you connect that processor's output to PutHDFS as explained here:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hadoop.PutHDFS/index.ht...

You might find these templates useful to help get you started:

https://github.com/hortonworks-gallery/nifi-templates/tree/master/templates

Thanks

Joe

View solution in original post

Re: how can I send send JMS to HDFS using nifi??

for download nifi I can do it on windows or linux? thanks for answer

Re: how can I send send JMS to HDFS using nifi??

Mentor

@vijay patel you can run nifi on anything that can run jvm.

Re: how can I send send JMS to HDFS using nifi??

Rising Star

You may also want to consider that JMS messages tend to be quite small, and before pushing to HDFS it may help to merge together many JMS messages so that you don't have a huge number of files in HDFS. You could use the MergeContent Processor to accomplish this.

Re: how can I send send JMS to HDFS using nifi??

Thanks mpayne

Re: how can I send send JMS to HDFS using nifi??

Guru

Configure ConsumeJMS processor, configure MergeContent processor to create suitable file sizes for HDFS, the use PutHDFS processor to write to HDFS directly.