Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Integrating custom logs and alerts into cloudera manager/hue

Integrating custom logs and alerts into cloudera manager/hue

Explorer

Hi.

As part of my setup, I have a number of scripts for input ingestion and oozie tasks, which as part of their processing do their fair share of info/warn/error logging in languages such as python,java, R.

 

I would like to have some centralized way to check out the errors on my pipeline, but would like a lightweight solution (i.e. I know splunk but afraid it may be an overkill for this). Is there some simpler solution, where I could leverage cloudera manager to analyze my applicational logs ? (notice that some of the ingestion scripts running on edge nodes, may even be simple "cron'ed" scripts and not integrated in the hadoop pipeline..

 

Even if I used flume to ingest logs, which would be the "lightest" application where I could analyze them  ?

 

thanks