Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Http-source flume into hdfs is not working​.? What could be hindering?

Highlighted

Http-source flume into hdfs is not working​.? What could be hindering?

Explorer

I have configured an agent with http-source source and sink as hdfs. I can now see the agent is working without any error. However, when i try to do a get or post request in python i dont get anything in hdfs. May be i am doing it wrong. Here is my agent configuration:

########## HTTP AGENT ########## 
HttpAgent.sources = http-source
HttpAgent.channels = MemChannel
HttpAgent.sinks = HDFS

#  agent SOURCES config
###############################
HttpAgent.sources.http-source.type = http
HttpAgent.sources.http-source.port = 8989
#HttpAgent.sources.http-source.handler = org.example.rest.RestHandler
#org.apache.flume.source.http.JSONHandler
HttpAgent.sources.http-source.bind     = 0.0.0.0
# FMI agent CHANNEL config
###############################
HttpAgent.channels.MemChannel.type = memory
HttpAgent.channels.MemChannel.capacity = 1000
HttpAgent.channels.MemChannel.transactionCapacity = 100

# agent SINK config
###############################
HttpAgent.sinks.HDFS.type = hdfs
HttpAgent.sinks.HDFS.hdfs.path=hdfs://hdp.localdomain:8020/data/input
HttpAgent.sinks.HDFS.hdfs.rollInterval=0
HttpAgent.sinks.HDFS.hdfs.rollSize=10485760
HttpAgent.sinks.HDFS.hdfs.rollCount=1

# Binding to the channel
###############################
HttpAgent.sources.http-source.channels=MemChannel
HttpAgent.sinks.HDFS.channel=MemChannel

At least the log shows http-source started. Moreover i know my sink path is also correct sink i used it to ingest twitter data and also netcat data. Now after i have restarted the agent in ambari, i opened a python console and did some get requests.

>>> requests.get('http://data.fmi.fi/fmi-apikey/14120224-39e0-4126-a9a1-ba4a1113c589/wfs?request=getFeature&storedy_id=fmi::forecast::hirlam::surface::point::multipointcoverage')
<Response [200]>

The response is 200. i event do the request with the text also but i dont know how it works. I dont know if i am doing it the right way. Your help will be highly appreciated.

  1. @grant
2 REPLIES 2

Re: Http-source flume into hdfs is not working​.? What could be hindering?

Master Collaborator

Can you post the flume agent log?

Re: Http-source flume into hdfs is not working​.? What could be hindering?

Explorer

I am trying to run a similar flume agent with HTTP source on the VM installed on my local computer. Not sure if the path and port is correct. When I ran my agent like this:

./flume-ng agent -n HttpAgent -c conf -f flume.conf
it is stuck at the point http-source started. Not sure what's happening and not able to see the flume logs either!

HttpAgent.sinks.hdfs-sink.hdfs.path = hdfs://quickstart.cloudera:8020/user/cloudera/output/http_flume
Also if you can explain what will my output file be like when this works?

Any suggestions?

Don't have an account?
Coming from Hortonworks? Activate your account here