Member since
07-09-2024
2
Posts
1
Kudos Received
0
Solutions
09-05-2024
12:04 AM
1 Kudo
Hi @RangaReddy Thanks for looking into my question. change the data type of the column to match the actual data format - I tried passing binary in catalog but had no luck. convert the data to a string format - It will result in data manipulation on HBase which is not practically a possible solution for us. Also, data size is somewhere around 50-60 TB. I am looking for an option where we can directly read those hexadecimal escape sequences(i.e., ReportV10\x00\x00\x00\x00\x02\x02\x02) as it is in my spark dataframe. Let me know if you need further clarity or information, we can setup a meeting to discuss this. Regards, Ayush
... View more
07-09-2024
11:33 PM
In HBase, i have a column qualifier in which i have a data like below: ReportV10\x00\x00\x00\x00\x02\x02\x02 When i am reading this table from spark using shc connect, i am getting junk characters in result. Below is the piece of code i am using to read a HBase table: catalog='''{ "table":{"namespace":"db1","name":"tb1"}, "rowkey":"key", "columns":{ "rowkey":{"cf":"rowkey","col":"key","type":"string"}, "nf_hh0":{"cf":"nf","col":"hh0","type":"string"} } }''' df=spark.read.option("catalog",catalog).format("org.apache.spark.sql.execution.datasources.hbase").load() df.show(1,False) +------------------------------------------------+ | rowkey | nf_hh0 | +---------------------------+------------------- + |26273707950926220...|ReportV10�� | +---------------------------+--------------------+ Spark version: 2.3.2.3.1.0.319-3 HBase version: 2.0.2.3.1.0.319-3 Python version: 2.7.5 Question: Is there any way to read those hexadecimal escape sequences as it is in a dataframe.
... View more