Member since
09-28-2016
8
Posts
1
Kudos Received
0
Solutions
03-01-2018
03:06 AM
Hi, I'm using impala-2.7.0-cdh5.10.0, and I find execute round() in impala-shell may give unexpected result, while hive can give correct result. e.g. impala> select round(269586/334026 * 100, 2);
80.70999999999999
hive > select round(269586/334026 * 100, 2);
80.71 Is this a known issue?
... View more
Labels:
- Labels:
-
Apache Impala
02-09-2018
11:46 PM
Thanks for you reply. I try to SET MT_DOP=0; before compute stats and it works! The impalad does not crash any more althougth compute stats still fail due to incompataible schema.
... View more
02-09-2018
06:16 AM
Recently we found lots of impala daemon exit in our cluster when executing 'compute stats' for some table. How to reproduce:
Impala version: 2.7.0-cdh5.10.0
Create table col_str_int
create table sample.col_str_int(
s STRING,
i INT
) stored as parquet;
Generate Parquet File with incompatible schema
create table sample.col_str_str (
s string,
i string
) stored as parquet;
insert into sample.col_str_str values("some_str", "false");
Copy parquet file to table
hadoop fs -cp /user/hive/warehouse/sample.db/col_str_str/* /user/hive/warehouse/sample.db/col_str_int/
Compute Stats
refresh sample.col_str_int;
compute stats sample.col_str_int;
Here is the message before impalad exit:
Wrote minidump to /data1/impala/logs/minidumps/impalad/57d8e9ec-a075-5f0c-54dbe818-09c7889c.dmp
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x0000000000000000, pid=25400, tid=0x00007f4c2c458700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_121-b13) (build 1.8.0_121-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.121-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C 0x0000000000000000
#
# Core dump written. Default location: /var/lib/impala/core or core.25400
#
# An error report file with more information is saved as:
# /var/lib/impala/hs_err_pid25400.log
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
"impalad.node153-84-98-jylt.qiyi.hadoop.impala.log.INFO.20180209-203116.25400" 501L, 28262C
So I want to ask:
Is this a know issue, has it been fixed in latest version?
Is there any workaround to such problem?
... View more
Labels:
- Labels:
-
Apache Impala
06-23-2017
12:49 AM
Hi, is it possible to define some set options, like REQUEST_POOL in ~/.impalarc? So that user doesn't need to execute "SET REQUEST_POOL=xxx" when start the impala-shell.
... View more
Labels:
- Labels:
-
Apache Impala
06-19-2017
01:19 AM
Hi, We also found Increment as a very useful operation in production. It can reduce row number. So why increment is not supported, is it on the plan?
... View more
06-08-2017
08:56 PM
Not putting python script under /usr/bin, but my working directory. For example, If someone else I don't known who write a python script under home directory: [~]$ cat abc.py
#!/usr/bin/python print "test script" Now I cannot execute impala-shell anymore due to Import Error: [~]$ impala-shell
test script
'import site' failed; use -v for traceback
Traceback (most recent call last):
File "/usr/lib/impala-shell/impala_shell.py", line 24, in <module>
import getpass
File "/usr/lib64/python2.6/getpass.py", line 18, in <module>
import os, sys, warnings
File "/usr/lib64/python2.6/os.py", line 398, in <module>
import UserDict
File "/usr/lib64/python2.6/UserDict.py", line 82, in <module>
import _abcoll
File "/usr/lib64/python2.6/_abcoll.py", line 11, in <module>
from abc import ABCMeta, abstractmethod
ImportError: cannot import name ABCMeta I think impala-shell shall not include current directory in PYTHONPATH to avoid such error.
... View more
06-07-2017
11:00 PM
When we execute impala-shell command at a directory contains user defined python script, and the script name conflicts with python module imported by impala-shell, we will encounter an Import Error. How to reproduce: create abc.py $ touch abc.py execute impala-shell: $ impala-shell
'import site' failed; use -v for traceback
Traceback (most recent call last):
File "/usr/lib/impala-shell/impala_shell.py", line 24, in <module>
import getpass
File "/usr/lib64/python2.6/getpass.py", line 18, in <module>
import os, sys, warnings
File "/usr/lib64/python2.6/os.py", line 398, in <module>
import UserDict
File "/usr/lib64/python2.6/UserDict.py", line 82, in <module>
import _abcoll
File "/usr/lib64/python2.6/_abcoll.py", line 11, in <module>
from abc import ABCMeta, abstractmethod
ImportError: cannot import name ABCMeta
... View more
Labels:
- Labels:
-
Apache Impala
09-28-2016
03:11 AM
1 Kudo
I find impala will give wrong answer if the result of Hive UDF is used in group by statement. The impala version is: 2.7.0-cdh5-IMPALA_KUDU-cdh5 RELEASE. Here is the procedure to reproduce the error: impala> create table test_escape_group_by (s string);
impala> insert into table test_escape_group_by values("longstring"), ("short");
impala> select my_escape_string(s) as es from test_escape_group_by;
longstring
short
impala> select my_escape_string(s) as es from test_escape_group_by group by es;
shorttring
short We can see that the beginning part of 'longstring' is replaced by 'short'. Here is the definition of my_escape_string: public class MyEscapeString extends UDF
{
public Text evaluate(Text para) throws ParseException {
if ((null == para) || ("".equals(para.toString()))) {
return new Text("");
}
return new Text(para.toString().replace("\\", "\\\\").replace("\"", "\\\""));
}
} My Question: Is this a bug of impala, or how can I rewritten the Java UDF to avoid such errors.
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Impala