Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

ETL lookup with NiFi and HBase lookup service with multiple columns

Highlighted

ETL lookup with NiFi and HBase lookup service with multiple columns

New Contributor

Greeting to all NiFi users......

109991-lookup-record-processor.png

I'm using NiFi 1.5.0 right now. We have a HBase table with two columns i.e msisdn and emsisdn. We need to match msisdn value with flowfile's msisdn field and if the match will happen then needs to update flowfile's msisdn and original_msisdn filed value with Hbase emsisdn string value. It's a lookup actually.


Our lookup file is quite huge and that's in HDFS, near about 56 GB. We are using LookupRecord processor with HBase_1_1_2_RecordLookupService . The lookup file is quite sensitive and not possible to stage in local file system.


Can you please help me out if there is any better approach or processor in NiFi 1.5.0. Our flowfile is csv file and already converted into json with avro schema before entering into LookupRecord. 109930-image1.png

Don't have an account?
Coming from Hortonworks? Activate your account here