Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Reporting and Analysis:

Solved Go to solution

Reporting and Analysis:

Expert Contributor

Hi,

I am working on a RFP and looking for an answer to:

Ability to recalculate and alert when there are changes to historical data within a time period within your solution:

What I don't understand is we cannot modify the data in HDFS. Its immutable. So the change of historical data, does that applies?

Any help is highly appreciated.

Thanks,

Sujitha

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Reporting and Analysis:

Super Guru

@sujitha sanku Here are some thought. Your right data in HDFS is immutable; however, with hive acid and phoenix/hbase you are able to update data. There are internal workings without those products which allow to update data. However at the core data exist in hdfs is not truly updated. It gives the perception. Hence why there is such thing as major/minor compaction. Not going to go into too much detail on that. So if data is updated in hbase, you can use NiFi to detect when a record is changed and based on that create a alert. As for hive/acid I am not aware of similar functionality. However products at attunity have functionality for CDC on hadoop. I would reach out to them. if that is not possible them you can build functionality to do some change tracking. It would be a custom solution. again that is for hive.

View solution in original post

1 REPLY 1
Highlighted

Re: Reporting and Analysis:

Super Guru

@sujitha sanku Here are some thought. Your right data in HDFS is immutable; however, with hive acid and phoenix/hbase you are able to update data. There are internal workings without those products which allow to update data. However at the core data exist in hdfs is not truly updated. It gives the perception. Hence why there is such thing as major/minor compaction. Not going to go into too much detail on that. So if data is updated in hbase, you can use NiFi to detect when a record is changed and based on that create a alert. As for hive/acid I am not aware of similar functionality. However products at attunity have functionality for CDC on hadoop. I would reach out to them. if that is not possible them you can build functionality to do some change tracking. It would be a custom solution. again that is for hive.

View solution in original post

Don't have an account?
Coming from Hortonworks? Activate your account here