Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How can i post the response.text data to flume source?

Highlighted

How can i post the response.text data to flume source?

Explorer

I am able to post a simple array of json events to the flume http source. However, Now i am targeting to ingest the text from the get request of the api.

response=requests.get('http://data.fmi.fi/fmi-apikey/1put-your-key-here/wfs?request=describeStoredQueries&storedquery_id=fmi::observations::weather::multipointcoverage&place=helsinki')

The above will work only after u put a key there. which can be requested for free.

The response.text is some xml format(gml). I have tried json handler which works fine with the json handler. However, i also tried the org.apache.flume.source.http.HTTPSourceXMLHandler which does not even allow the connection. Any good idea how to post the response.text to the http source.

2 REPLIES 2

Re: How can i post the response.text data to flume source?

New Contributor

Change handler to blob handler if you want to send the whole response to Flume as is.

a1.sources.r1.type = http
a1.sources.r1.handler = org.apache.flume.sink.solr.morphline.BlobHandler
a1.sources.r1.port = 44444

Python code

import requests

flume_url = 'http://127.0.0.1:44444'
api_key = 'put your api key here'
source_url = 'http://data.fmi.fi/fmi-apikey/{0}/wfs?request=describeStoredQueries&storedquery_id=fmi::observations::weather::multipointcoverage&place=helsinki'.format(api_key)

response = requests.get(source_url)
requests.post(flume_url, data=response.text)

Re: How can i post the response.text data to flume source?

Explorer

@mhegedus Thank you. Yes changing the handler help. However, i had to use the unicode('utf-8') function for the text otherwise it was failing with error. Now i have to look for some python library to parse the values.

Don't have an account?
Coming from Hortonworks? Activate your account here