Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Falcon Reporting capabilities

Falcon Reporting capabilities

New Contributor

Does Falcon have reporting capabilities? Pre-built or ability to generate our own custom reports? For example, Files nearing expiration. Files that were purged last month or within 14 days etc

4 REPLIES 4
Highlighted

Re: Falcon Reporting capabilities

I've been searching for the same functionality, but I don't see it in Falcon 0.7. Looking at the source code, I see that replication statistics are saved, but not eviction statistics. I see counters being written in this file: https://github.com/apache/falcon/blob/master/replication/src/main/java/org/apache/falcon/replication..., but nothing similar in FeedEvictor.java

It would be a useful feature to send a JMS message before a file is evicted so that another process can be spawned in addition to a report of recently evicted files.

Highlighted

Re: Falcon Reporting capabilities

Rising Star

Falcon does not have reporting capabilities for listing files that are nearing expiration. If it is useful to receive an email with list of files that are purged through Retention job, Falcon has the capability to add this support. Please open a Jira.

Highlighted

Re: Falcon Reporting capabilities

Contributor

It's not 100% what you're looking for, but we do leverage the JMS messages Falcon sends upon completion of actions like processes and replication. This allowed us to build a job history for all of our jobs. We have a client that is capturing those messages and writing them to a pseudo-database that a web app puts into a table that our team can use for supporting and monitoring jobs. It's a handy use of JMS.

You could do some combination of that with smart logic: read the retention time from an entity, compare with the current date, and send alerts for jobs letting the team know how much time is left before eviction, replication, or the next instance of anything happens.

Example attached in pic.

686-screen-shot-2015-12-09-at-11707-pm.png

Highlighted

Re: Falcon Reporting capabilities

Mentor

@sraghavan has this been resolved? Can you post your solution or accept best answer?

Don't have an account?
Coming from Hortonworks? Activate your account here