Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to : HTTP Stream in flink

Solved Go to solution

How to : HTTP Stream in flink

Contributor

Which methods/objects does Flink provide in order to enable users to read HTTP streams?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: How to : HTTP Stream in flink

Contributor

@Robert Metzger,

I have prepared the Custom HTTPJSONStream reader and here is the attached code.

loganalytics.txt-> This is the main Class which fetches the data from the custom source. In our case its JSON via HTTP url.

httpjsonstream.txt -> This class implements the SourceFunction and provides a SourceContext of custom-type FlinkJSONObject

flinkjsonobject.txt -> This class uses java.net.* and javax.json.* to connect to provided URL and get the content in JSON Object format.

The logic can be applied on the JSONObject to get the desired results.

Please suggest if any changes can be made.

7 REPLIES 7

Re: How to : HTTP Stream in flink

@Hemant Kumar

The best information on this topic can be found on flink site. https://flink.apache.org/news/2015/02/09/streaming...

https://github.com/dataartisans/flink-streaming-de...

Not sure if it helps to customize for HTTP

Re: How to : HTTP Stream in flink

Contributor

Thanks @Neeraj Sabharwal

For HTTP, I am in a state where I have to write a custom SourceFunction. Before I try it, I am exploring if there is a possibility to Handle HTTP Requests by using any of the existing StreamExecutionEnvironment functions/objects.

I am also trying to handle HTTP Stream for Flink using NiFi Connector as mentioned here

I hope either of these will help me in achieving desired results. :)

Re: How to : HTTP Stream in flink

Re: How to : HTTP Stream in flink

New Contributor

Sorry for the late reply. As I said in the other thread, I would recommend using a SourceFunction here. Feel free to post your code here as well, I can review it and give some hints for improving it.

Re: How to : HTTP Stream in flink

Contributor

@Robert Metzger,

I have prepared the Custom HTTPJSONStream reader and here is the attached code.

loganalytics.txt-> This is the main Class which fetches the data from the custom source. In our case its JSON via HTTP url.

httpjsonstream.txt -> This class implements the SourceFunction and provides a SourceContext of custom-type FlinkJSONObject

flinkjsonobject.txt -> This class uses java.net.* and javax.json.* to connect to provided URL and get the content in JSON Object format.

The logic can be applied on the JSONObject to get the desired results.

Please suggest if any changes can be made.

Re: How to : HTTP Stream in flink

New Contributor

Thank you for posting the code here @Hemant Kumar. Sorry for the delay, I've now looked at and it looks good. If you want to optimize the code, I would try to move the code from the FlinkJSONObject constructor into the HTTPJSONStream class. It seems that the do/while loop is actually controlled from within the FlinkJSONObject.

Re: How to : HTTP Stream in flink

Contributor

Thanks a lot @Robert Metzger

Don't have an account?
Coming from Hortonworks? Activate your account here