Support Questions
Find answers, ask questions, and share your expertise

How to configure Flume to listen a web api http petitions

How to configure Flume to listen a web api http petitions

New Contributor

I have built an api web application, which is published on IIS Server, I am trying to configure Apache Flume to listen that web api and to save the response of http petitions in HDFS, this is the post method that I need to listen:

    public IEnumerable<Data> obtenerValores(arguments arg)
        Random rdm = new Random();

        int ano = arg.ano;
        int rdmInt;
        decimal rdmDecimal;

        int anoActual = DateTime.Now.Year;
        int mesActual = DateTime.Now.Month;

        List<Data> ano_mes_sales = new List<Data>();

        while (ano <= anoActual)
            int mes = 1;
            while ((anoActual == ano && mes <= mesActual) || (ano < anoActual && mes <= 12))
                rdmInt = rdm.Next();
                rdmDecimal = (decimal)rdm.NextDouble();
                Data anoMesSales = new Data(ano, mes,(rdmInt * rdmDecimal));

        return ano_mes_sales;

Flume is running over a VMware Virtual Machine CentOs, this is my attempt to configure flume to listen that application:

# Sources, channels, and sinks are defined per # agent name, in this case 'tier1'.
a1.sources  = source1
a1.channels = channel1
a1.sinks    = sink1
a1.sources.source1.interceptors = i1 i2 
a1.sources.source1.interceptors.i1.type = host
a1.sources.source1.interceptors.i1.preserveExisting = false
a1.sources.source1.interceptors.i1.hostHeader = host
a1.sources.source1.interceptors.i2.type = timestamp

# For each source, channel, and sink, set # standard properties.
a1.sources.source1.type     = org.apache.flume.source.http.HTTPSource
a1.sources.source1.bind     =
a1.sources.source1.port     = 80

# JSONHandler is the default for the httpsource # 
a1.sources.source1.handler = org.apache.flume.source.http.JSONHandler
a1.sources.source1.channels = channel1
a1.channels.channel1.type   = memory
a1.sinks.sink1.type         = hdfs
a1.sinks.sink1.hdfs.path = /monthSales
a1.sinks.sink1.hdfs.filePrefix = event-file-prefix-
a1.sinks.sink1.hdfs.round = false      = channel1

# Other properties are specific to each type of # source, channel, or sink. In this case, we # specify the capacity of the memory channel.
a1.channels.channel1.capacity = 1000 

I am using curl to post, here is my attempt:

curl -X POST -H 'Content-Type: application/json; charset=UTF-8' -d '[{"ano":"2010"}]'

I only get this error:


My question are, which is the right way to configure flume to listen http petitions to my web api, what I am missing?

Don't have an account?