Special characters are displayed as question mark (?).

Question

We have a delimited flatfile which has special characters ingested into HDFS. We have created a Hive External table on top of the HDFS path and visualized the data in Hue & Beeline. But the special characters are displayed as question mark (?). Please advise the solution.

smruti · Accepted Answer

we tried replicating the issue with the data shared by @Shivakuk . Left/Right Single/Double Quotation Mark(smart quotes) in the text did not show up correctly and got converted to ? . I was able to fix this issue by changing the LC_CTYPE from "UTF-8" to "en_US.UTF-8".

Check "locale" command output:

# locale
LANG=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See what your LC_CTYPE read.
If it does not read "en_US.UTF-8", do the following:

vi ~/.bash_profile

Add the following two lines at the bottom:
+++
LC_CTYPE=en_US.UTF-8
export LC_CTYPE
+++

Save the file, and source it for it to take effect:

#source ~/.bash_profile

Now connect to beeline, and see if the data show up correctly.

asish · Answer

@Shivakuk Did you use Ranger masking for that column? Provide the below output  show create table <tablename>

Sign up

Please use your Cloudera account.

Login to the community

Please use your Cloudera account.