Support Questions

Find answers, ask questions, and share your expertise

What should be the data handler for the flume http agent?



I am doing a post request to the http source of flume agent. I am able to do a post request and post the data to the flume agent. However, the file is empty if i push some string. But i am able to send data as a list of json events as given in the code(that mean the json handler is used). This means there is not a proper handler flume data handler.Th handler works perfectly for any type of text data(org.apache.flume.sink.solr.morphline.BlobHandler) in python but it does not help when using this java method to make the request. Here is the method. It takes a string input.

public static void sendGet(String ar) throws Exception{String url2="";//The url of the flume http source
        URL obj2 = new URL(url2);HttpURLConnection con2 =(HttpURLConnection) obj2.openConnection();// create a connection object       
con2.setRequestMethod("POST");System.out.println("\nSending 'post' request to URL : "+ url2);//System.out.println("Response Code : "+ responseCode2);//You need to call teh output stream of the connection object and write to it.
        con2.connect();OutputStream os =con2.getOutputStream();
        os.write("hello".getBytes());// this did not write. only created empty file.
        os.write(ar.getBytes());//With he default json handler i was able to write the below line to hdfs via flume http source.    
    os.write("[{body:ok}]".getBytes());System.out.println("okie");if(con2.getResponseCode()!=200){System.out.println("error founded");System.out.println(con2.getResponseCode());}//in the end to close the connection. 


Do you want to write a program which continuously polls a web server and writes that data to HDFS? If this is the case then you could add dependency of groupId: org.apache.hadoop, artifactId: hadoop-client and call append on HDFS FileSystem api directly (without the use of flume). Different approach would be to start an embedded flume agent inside your application. That way you do not have to setup the flume source but you could directly put events to the flume channel.


Do you have any exceptions when you run the above client? Try running the Flume agent with the extra option