question Re: Is it possible to ingest data into kafka from REST API calls? in Archives of Support Questions (Read Only)

Is it possible to ingest data into kafka from REST API calls?

nsabharwal — Sat, 12 Dec 2015 10:12:57 GMT

HDP 2.3.2

We have data sitting on prem and we are planning to ingest data into Kafka using REST calls? Kafka will be running in the cloud.

Re: Is it possible to ingest data into kafka from REST API calls?

nsabharwal — Sat, 12 Dec 2015 10:14:53 GMT

Nifi is not the option yet.

Re: Is it possible to ingest data into kafka from REST API calls?

aervits — Sat, 12 Dec 2015 23:55:47 GMT

@Neeraj Sabharwal nifi example working with csv does that https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates

Re: Is it possible to ingest data into kafka from REST API calls?

aervits — Sat, 12 Dec 2015 23:56:54 GMT

@Neeraj Sabharwal OK I see your nifi comment

Re: Is it possible to ingest data into kafka from REST API calls?

nsabharwal — Sun, 13 Dec 2015 01:03:49 GMT

@Guilherme Braccialli

Guil shared some ideas on using Flume.

Re: Is it possible to ingest data into kafka from REST API calls?

aervits — Wed, 03 Feb 2016 23:37:28 GMT

please close @Neeraj Sabharwal

Re: Is it possible to ingest data into kafka from REST API calls?

nsabharwal — Wed, 03 Feb 2016 23:40:29 GMT

@Artem Ervits

This is the answer that I got from the client https://github.com/confluentinc/kafka-rest

That's the answer without buying HDF.

Re: Is it possible to ingest data into kafka from REST API calls?

nsabharwal — Wed, 03 Feb 2016 23:41:07 GMT

@Artem Ervits updated...feel free to accept my answer

Re: Is it possible to ingest data into kafka from REST API calls?

sbomma — Tue, 20 Sep 2016 00:08:09 GMT

Does Confluent Inc Support cost extra if someone wants to install this in Production? Is there a hortonworks out of the box solution to get RESTful APIs data to be ingested into HDFS.

Re: Is it possible to ingest data into kafka from REST API calls?

pravat_sutar01 — Mon, 22 Jul 2019 09:12:43 GMT

Nifi and Kafka example for pulling REST API end point data -

https://dzone.com/articles/real-time-stock-processing-with-apache-nifi-and-ap

Re: Is it possible to ingest data into kafka from REST API calls?

TimothySpann — Thu, 23 Jan 2020 19:46:06 GMT

Okay so I wrote an example nifi process to do it

https://www.datainmotion.dev/2020/01/flank-stack-nifi-processor-for-kafka.html