Support Questions
Find answers, ask questions, and share your expertise

Spark streaming log access from unix

Spark streaming log access from unix

New Contributor

I am running the spark streaming job every minutes, and able get job info like Batch Time,Input Size,Scheduling Delay,Processing Time,Total Delay from below spark streaming link.

http://hostName:8088/proxy/application_1510017956315_42239/streaming/

Below is the spark streaming image :


46381-spark-streaming-gui.png

I need to write each stream info into NOSQL/Database ( Batch Time,Input Size,Scheduling Delay,Processing Time,Total Delay ). I need to read spark streaming log information from unix , so that i can write into NoSQL OR other database.


Please suggest, how to read the log info from unix.

Thanks,

Saravanan

1 REPLY 1

Re: Spark streaming log access from unix

Super Collaborator

Hi @Saravanan Nagarajan,

you can get the all the above metrics form a REST API and once you got the content you can process them and store in SQL/NOSQL or File based store.

the end point for the REST call would be http://<host>:4040/api/v1/applications/[app-id]/streaming/statistics

this API provides the information about : Statistics for the streaming context ( in fact you can keep this process inside the running job by querying the same context)

more on this REST calls can be found here

Hope this helps !!