Reply
Highlighted
Explorer
Posts: 15
Registered: ‎06-09-2015

Not able to store data from ActiveMQ to HDFS (through Flume), which logs to check?

I started my Flume services, check Flume logs in /var/logs/flume-ng, agent started, everything seems fine. However, the data in ActiveMQ is not stored in my HDFS. May I know what other logs I can check to see where went wrong?

 

Also, do we need to specify the activeMQ topic in the Flume configuration? The example from Flume user guide does not have a topic.

 

a1.sources = r1
a1.channels = c1
a1.sources.r1.type = jms
a1.sources.r1.channels = c1
a1.sources.r1.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory
a1.sources.r1.connectionFactory = GenericConnectionFactory
a1.sources.r1.providerURL = tcp://mqserver:61616
a1.sources.r1.destinationName = BUSINESS_DATA
a1.sources.r1.destinationType = QUEUE

 

Cloudera Employee
Posts: 255
Registered: ‎01-09-2014

Re: Not able to store data from ActiveMQ to HDFS (through Flume), which logs to check?

destinationName and destinationQueue determine where to pull messages from in activemq. Is "BUSINESS_DATA" your topic that you are pulling from? Also, you may want to change destinationType to TOPIC instead of QUEUE.

You should see error messages in the flume-cmf log if it is having problems connecting to flume.

-pd
Explorer
Posts: 15
Registered: ‎06-09-2015

Re: Not able to store data from ActiveMQ to HDFS (through Flume), which logs to check?

Yes, the topic is causing the problem. Now I am able to see data inserted to hdfs after changing the topic.

 

If I have more than 1 topic in activemq, how do I specify it in the flume configuration?

Announcements
New solutions