Support Questions

Find answers, ask questions, and share your expertise

How to perform lookup operation in spark dataframe

avatar

Based on the columns in spark dataframe need to do a lookup on another huge HBASE table. Is there any efficient way available to perform lookup operation in Spark Dataframe

1 REPLY 1

avatar
Super Collaborator

I think if one of the columns in the dataframe is the key of the HBase table, the lookup will be very efficient; that is, if there is a bottleneck, I don't believe the bottleneck will be the lookup.