question Re: Use REST API as source in Flume in Archives of Support Questions (Read Only)

Use REST API as source in Flume

Vinod369 — Fri, 16 Sep 2022 10:23:57 GMT

I am trying to read the data from SQL Server using REST API as a Flume source with JSON format.

Not sure which Flume source I can use here (may be HTTP) ?

If it is HTTP then where I should mention the API URL(http://api.xxxx.com.xxxx) in Flume source property ?

Please advice.

Re: Use REST API as source in Flume

pdvorak — Tue, 07 Jun 2016 23:23:39 GMT

The flume http source is for creating a REST API that you can post data to from the upstream sender. If you are looking for a source that will consume from SQL server via accessing the SQL server API, you'll need to write a custom source for that, or possibly try this: https://github.com/keedio/flume-ng-sql-source

Additionally, if you don't need real time processing, you may want to consider using sqoop to import data via batch processing, and it can handle incremental updates.

-pd