Member since
03-22-2017
10
Posts
0
Kudos Received
0
Solutions
05-08-2017
09:35 AM
I have made a test. I create a text file file content in Unicode UTF-8: > cat test.csv
björn,alvägen
> file test.csv
test.csv: UTF-8 Unicode text
And create a table reading from that csv file; hive> CREATE EXTERNAL TABLE test (
> column1 String,
> column2 String
> )
> ROW FORMAT DELIMITED FIELDS TERMINATED BY ','
> LOCATION '/user/myuser/test/';
hive> select * from test;
OK
björn alvägen
Time taken: 0.058 seconds, Fetched: 1 row(s)
But in HIVE VIEW: test.column1 test.column2
bj?rn alv?gen Similar issue happens for Zeppelin. So, it is not a problem with encoding of source file @Jay SenSharma, @Umair Khan @Shyam Sunder Rai as I have source file in URF-8 and Hive can query data and display it in correct UTF-8. It must be an issue with HIVE VIEW and Zeppelin
... View more
05-08-2017
09:09 AM
We have checked and the encoding for the file is UTF-8. Please note that if we use hive command line interface to SELECT *, the text display correctly in the terminal. But in Hive View and Zeppelin, the text display wrong. I have also tested with different browser and different encodings, but Hive view just cannot rendered if not using Unicode UTF-8.
... View more
05-08-2017
09:05 AM
We have checked and the encoding for the file is UTF-8. Please note that if we use hive command line interface to SELECT *, the text display correctly in the terminal. But in Hive View and Zeppelin, the text display wrong. So, it might not be an issue of Hive but Hive view/Zeppelin.
... View more
05-05-2017
01:23 PM
I am using HDP 2.6 with Ambari 2.5. We have such problem both in Zeppelin and Hive-view.
... View more
05-05-2017
12:49 PM
Hi All, I have some string text in UTF-8 in Hive tables. Querying using HiveQL from terminal or using SparkQL using spark-shell produces the text correctly, no encoding error: >> select address from customerinfo limit 2;
address
----------------
Brahegatan 50
Envägen 12 However, when I query the data using Hive View in Ambari, the result are not display correctly. select address from customerinfo limit 2;
address
----------------
Brahegatan 50
Env?gen 12 Similar issue happens when querying with Zeppelin using SparkQL. Is this a bug in Hive view or Ambari?
Do you know what is the cause of the problem and the solution? BR, /Nhan
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
04-03-2017
07:12 AM
That explains because we have Hive 1.2.1000. Thanks @ssanthosh!
... View more
03-24-2017
08:06 AM
Thanks @ssanthosh. It works! It is wierd that some string functions are allowed (substr, regexp_replace) and some are not (replace_()) in the "custom" masking option. Is there any document about it?
... View more
03-22-2017
03:12 PM
Hi, I am using Apache Ranger to specify masking policies to hide sensitive information from Hadoop users. I manage to use custom masking function like mask, mask_last_n, mask_first_n,.... Now I want to create a custom masking rule for the email address column so that all email name will be replaced by "email". For example: john_doe@gmail.com will become email@gmail.com. The custom masking expression using UDF look like this: replace({col}, substr({col}, 0, instr({col},"@")), "email") But when I use this custom masking expression, Hive query to that field always cause error: java.lang.Exception: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Line 0:-1 Invalid function 'replace'
java.lang.Exception: org.apache.hive.service.cli.HiveSQLException: Error while compiling statement: FAILED: SemanticException Line 0:-1 Invalid function 'replace'
at org.apache.ambari.view.hive2.resources.jobs.JobService.getOne(JobService.java:141)
at sun.reflect.GeneratedMethodAccessor1289.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
...
This Ranger document says when select Custom masking type: Custom – Specify a custom masked value or expression. Custom masking can use any valid Hive UDF (Hive that returns the same data type as the data type in the column being masked).
Masking conditions are evaluated in the order listed in the policy. The condition at the top of the Masking Conditions list is applied first, then the second, then the third, and so on. As said here , replace is a Hive UDF function which return string similar to the data type of the column. But still it is not possible to use replace. Can someone help me on how to use Hive UDF function with Custom masking type? Or if you can help me with masking email address, that would be welcome too.
... View more
Labels:
- Labels:
-
Apache Ambari
-
Apache Hive
-
Apache Ranger