Support Questions

Find answers, ask questions, and share your expertise

Hbase filter query using pyspark

avatar
Explorer

I am trying to pull the records from the Hbase based on the row key in pyspark using the following details

 

pyspark.JPG

but the records which is returning showing all the records (i,e without filter), can you please help me to resolving the issue

 

1 ACCEPTED SOLUTION

avatar
Master Collaborator

Hi @Taries 

I hope you are doing good. Do you need any further help on this issue. If above solutions is helped in your case please accept the Solution. It will help for others. 

View solution in original post

5 REPLIES 5

avatar
Community Manager

@Taries, Welcome to our community! To help you get the best possible answer, I have tagged in our HBase and Spark experts  @smdas  @RangaReddy who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.



Regards,

Vidya Sargur,
Community Manager


Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
Learn more about the Cloudera Community:

avatar
Master Collaborator

Hi @Taries 

You need to use the following two parameters to apply filter.

hbase.spark.query.timerange.start

hbase.spark.query.timerange.end

Reference:

https://github.com/apache/hbase-connectors/blob/307607cf7287084b3ce49cdd96d094e2ede9363a/spark/hbase...

avatar
Explorer

Taries_0-1706710173347.png

 

with the above given options, it is only taking the timerange start and end , not considering the rowkey passed along with it. 

avatar
Master Collaborator

Hi @Taries 

As I mentioned previously, only the hbase.spark.query.timerange parameter can be used for filtering data during read. The hbase.spark.scan parameter wouldn't be set for this purpose.

To filter the data after reading, you can apply a Spark WHERE or filter clause with your desired conditions.

 

avatar
Master Collaborator

Hi @Taries 

I hope you are doing good. Do you need any further help on this issue. If above solutions is helped in your case please accept the Solution. It will help for others.