Member since
06-20-2016
3
Posts
4
Kudos Received
0
Solutions
10-31-2016
01:13 PM
3 Kudos
Repo Description Ephemeral means that we do not know the state of Spark before, during, or after the invocation, we just care that the job was submitted, executed and results came back, no state of any type is kept. Data, or reference to data, is passed in the invocation, and the results captured and stored in the flowfile. The best use case for this processor is when you want to launch ephemeral jobs for processing in Spark with a predictable start and finish cycles, rather than long running jobs. Since the processor will launch-and-wait for spark to finish, it means a thread will be allocated to wait until spark returns, no interaction happens while the Spark job is running. Repo Info Github Repo URL https://github.com/diegobaez/PUBLIC/tree/master/NiFi-SnapSpark Github account name diegobaez Repo name NiFi-SnapSpark
... View more
Labels:
09-20-2016
07:31 PM
1 Kudo
The requirement is to have HBASE notify (PUSH) to NiFi when a piece of data has been changed, rather than use the polling mechanism currently implemented in NiFi. Client has implemented a custom in-house solution using HBASE coprocessor, but would like to replace this code with NiFi. Polling does not work well in an environment when data can change rapidly or when data is updated at irregular intervals. Ideal solution would be to "publish-and forget" an update via Kafka (or another method) in a manner which does not stop HBASE processing like a tigger would or has an impact of performance of HBASE.
... View more