question Re: Hbase filter query using pyspark in Support Questions

Hbase filter query using pyspark

Taries — Wed, 31 Jan 2024 05:22:24 GMT

I am trying to pull the records from the Hbase based on the row key in pyspark using the following details

but the records which is returning showing all the records (i,e without filter), can you please help me to resolving the issue

Re: Hbase filter query using pyspark

VidyaSargur — Wed, 31 Jan 2024 07:02:11 GMT

@Taries, Welcome to our community! To help you get the best possible answer, I have tagged in our HBase and Spark experts @smdas @RangaReddy who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

Re: Hbase filter query using pyspark

RangaReddy — Wed, 31 Jan 2024 08:34:00 GMT

Hi @Taries

You need to use the following two parameters to apply filter.

hbase.spark.query.timerange.start

hbase.spark.query.timerange.end

Reference:

https://github.com/apache/hbase-connectors/blob/307607cf7287084b3ce49cdd96d094e2ede9363a/spark/hbase-spark/src/main/scala/org/apache/hadoop/hbase/spark/datasources/HBaseSparkConf.scala#L65

Re: Hbase filter query using pyspark

Taries — Wed, 31 Jan 2024 14:09:35 GMT

with the above given options, it is only taking the timerange start and end , not considering the rowkey passed along with it.

Re: Hbase filter query using pyspark

RangaReddy — Wed, 31 Jan 2024 14:59:57 GMT

Hi @Taries

As I mentioned previously, only the hbase.spark.query.timerange parameter can be used for filtering data during read. The hbase.spark.scan parameter wouldn't be set for this purpose.

To filter the data after reading, you can apply a Spark WHERE or filter clause with your desired conditions.

Re: Hbase filter query using pyspark

RangaReddy — Sun, 04 Feb 2024 15:46:28 GMT

Hi @Taries

I hope you are doing good. Do you need any further help on this issue. If above solutions is helped in your case please accept the Solution. It will help for others.