Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Use REST API as source in Flume

SOLVED Go to solution

Use REST API as source in Flume

Contributor

I am trying to read the data from SQL Server using REST API as a Flume source with JSON format.

Not sure which Flume source I can use here (may be HTTP) ?

 

If it is HTTP then where I should mention the API URL(http://api.xxxx.com.xxxx) in Flume source property ?

Please advice.

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Use REST API as source in Flume

Super Collaborator
The flume http source is for creating a REST API that you can post data to from the upstream sender. If you are looking for a source that will consume from SQL server via accessing the SQL server API, you'll need to write a custom source for that, or possibly try this: https://github.com/keedio/flume-ng-sql-source

Additionally, if you don't need real time processing, you may want to consider using sqoop to import data via batch processing, and it can handle incremental updates.

-pd
1 REPLY 1

Re: Use REST API as source in Flume

Super Collaborator
The flume http source is for creating a REST API that you can post data to from the upstream sender. If you are looking for a source that will consume from SQL server via accessing the SQL server API, you'll need to write a custom source for that, or possibly try this: https://github.com/keedio/flume-ng-sql-source

Additionally, if you don't need real time processing, you may want to consider using sqoop to import data via batch processing, and it can handle incremental updates.

-pd