question Re: I want to use source as JMS and sink to Kafka using flume, what kinds of configuration need to do? in Archives of Support Questions (Read Only)

I want to use source as JMS and sink to Kafka using flume, what kinds of configuration need to do?

pansheriya_vija — Wed, 20 Jan 2016 23:40:52 GMT

Re: I want to use source as JMS and sink to Kafka using flume, what kinds of configuration need to do?

aervits — Wed, 20 Jan 2016 23:47:19 GMT

@vijay patel

you need JMS Source

also need Kafka Sink

and a channel of your choice

Re: I want to use source as JMS and sink to Kafka using flume, what kinds of configuration need to do?

pansheriya_vija — Thu, 21 Jan 2016 00:02:04 GMT

HI Ervits Thanks for quick reply...

can you please show me flume.conf file configuration here if you dont mind

Re: I want to use source as JMS and sink to Kafka using flume, what kinds of configuration need to do?

aervits — Thu, 21 Jan 2016 00:04:14 GMT

there are examples at the links I provided

a1.sources = r1
a1.channels = c1
a1.sources.r1.type = jms
a1.sources.r1.channels = c1
a1.sources.r1.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory
a1.sources.r1.connectionFactory = GenericConnectionFactory
a1.sources.r1.providerURL = tcp://mqserver:61616
a1.sources.r1.destinationName = BUSINESS_DATA
a1.sources.r1.destinationType = QUEUE
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = mytopic
a1.sinks.k1.brokerList = localhost:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20
a1.sinks.k1.channel = c1

again you also need to add a channel, there are tons of examples on the same site. @vijay patel

Re: I want to use source as JMS and sink to Kafka using flume, what kinds of configuration need to do?

pansheriya_vija — Thu, 21 Jan 2016 00:06:28 GMT

Thanks you so much

Re: I want to use source as JMS and sink to Kafka using flume, what kinds of configuration need to do?

pansheriya_vija — Thu, 21 Jan 2016 00:08:51 GMT

Is it any jar file or any properties need to be changed above conf file?

Re: I want to use source as JMS and sink to Kafka using flume, what kinds of configuration need to do?

aervits — Thu, 21 Jan 2016 00:12:02 GMT

@vijay patel I suggest doing a tutorial on flume, there are no other jars necessary.

Re: I want to use source as JMS and sink to Kafka using flume, what kinds of configuration need to do?

pansheriya_vija — Thu, 21 Jan 2016 00:13:30 GMT

I need to send data JMS-->kafka and kafka to HDFS using flume can you advise me what is the best option and conf file looks like?

Re: I want to use source as JMS and sink to Kafka using flume, what kinds of configuration need to do?

aervits — Thu, 21 Jan 2016 00:18:40 GMT

@vijay patel you need to open a new thread as this is different requirement from the original question. This can become messy as you will have to chain Flume agents. The only way to chain the agents is to use Avro RPC. We recommend you look at HDF powered by Apache Nifi. It will be a lot simpler to do.