Support Questions
Find answers, ask questions, and share your expertise

How can i post the response.text data to flume source?


I am able to post a simple array of json events to the flume http source. However, Now i am targeting to ingest the text from the get request of the api.


The above will work only after u put a key there. which can be requested for free.

The response.text is some xml format(gml). I have tried json handler which works fine with the json handler. However, i also tried the org.apache.flume.source.http.HTTPSourceXMLHandler which does not even allow the connection. Any good idea how to post the response.text to the http source.



Change handler to blob handler if you want to send the whole response to Flume as is.

a1.sources.r1.type = http
a1.sources.r1.handler = org.apache.flume.sink.solr.morphline.BlobHandler
a1.sources.r1.port = 44444

Python code

import requests

flume_url = ''
api_key = 'put your api key here'
source_url = '{0}/wfs?request=describeStoredQueries&storedquery_id=fmi::observations::weather::multipointcoverage&place=helsinki'.format(api_key)

response = requests.get(source_url), data=response.text)


@mhegedus Thank you. Yes changing the handler help. However, i had to use the unicode('utf-8') function for the text otherwise it was failing with error. Now i have to look for some python library to parse the values.