Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

I want to use source as JMS and sink to Kafka using flume, what kinds of configuration need to do?

avatar
Contributor
 
1 ACCEPTED SOLUTION

avatar
Master Mentor

there are examples at the links I provided

a1.sources = r1
a1.channels = c1
a1.sources.r1.type = jms
a1.sources.r1.channels = c1
a1.sources.r1.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory
a1.sources.r1.connectionFactory = GenericConnectionFactory
a1.sources.r1.providerURL = tcp://mqserver:61616
a1.sources.r1.destinationName = BUSINESS_DATA
a1.sources.r1.destinationType = QUEUE
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = mytopic
a1.sinks.k1.brokerList = localhost:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20
a1.sinks.k1.channel = c1 

again you also need to add a channel, there are tons of examples on the same site. @vijay patel

View solution in original post

8 REPLIES 8

avatar
Master Mentor

@vijay patel

you need JMS Source

also need Kafka Sink

and a channel of your choice

avatar
Contributor

HI Ervits Thanks for quick reply...

can you please show me flume.conf file configuration here if you dont mind

avatar
Master Mentor

there are examples at the links I provided

a1.sources = r1
a1.channels = c1
a1.sources.r1.type = jms
a1.sources.r1.channels = c1
a1.sources.r1.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory
a1.sources.r1.connectionFactory = GenericConnectionFactory
a1.sources.r1.providerURL = tcp://mqserver:61616
a1.sources.r1.destinationName = BUSINESS_DATA
a1.sources.r1.destinationType = QUEUE
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = mytopic
a1.sinks.k1.brokerList = localhost:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20
a1.sinks.k1.channel = c1 

again you also need to add a channel, there are tons of examples on the same site. @vijay patel

avatar
Contributor

Thanks you so much

avatar
Contributor

Is it any jar file or any properties need to be changed above conf file?

avatar
Master Mentor

@vijay patel I suggest doing a tutorial on flume, there are no other jars necessary.

avatar
Contributor

I need to send data JMS-->kafka and kafka to HDFS using flume can you advise me what is the best option and conf file looks like?

avatar
Master Mentor

@vijay patel you need to open a new thread as this is different requirement from the original question. This can become messy as you will have to chain Flume agents. The only way to chain the agents is to use Avro RPC. We recommend you look at HDF powered by Apache Nifi. It will be a lot simpler to do.