Member since
06-28-2016
55
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
950 | 07-22-2016 02:08 PM |
09-16-2019
09:21 AM
@tmater Thank you for the update. What if some one alters the table, without knowing this fact that it would not work in impala. If we have a huge data set(historical data) that the user will be unable to view in impala. what is best way to overcome this. 1.Is there a way we can edit the parquet files metadata in impala either thru spark/from command line ?. 2.we tried creating a temporary table with new schema and copying data from old table with Lateral view explode(on array/complex data type) in hive(manually with hive queries) and it is taking lot of our cluster resource and terminating without doing anything. 3.Even we tried to split the historical data(source) with date range and it is not helping us. I believe Lateral view explode is taking lot of resource on a historical data and causing to terminate. Notes: we have complex data type in our source table and so we are using Lateral view explode in hive Any work around either in spark/hive/impala anything that will help to deal with historical and current data in impala would be really helpful. Thank you for your help on this
... View more
09-16-2019
01:31 AM
1.We have hive table with below schema.
create external table precision_1 ( id String, prec decimal(7,2) ) partitioned by (date string) stored as parquet;
2.insert into precision_1 partition(date='2019-02-01') values ('1', 2.34);
3.insert into precision_1 partition(date='2019-02-01') values ('2', 3.45);
3. 'SELECT * from precision_1' works in both impala and Hive
4.We did following in Hive and invalidated metadata in impala:
ALTER TABLE precision_1 CHANGE prec prec decimal(9,6);
insert into precision_1 partition(date='2019-01-02') values ('1', 3.765894);
'SELECT * from precision_1' works only in hive but not in imala
In impala throws the below erorr.
Bad status for request TFetchResultsReq(fetchType=0, operationHandle=TOperationHandle(hasResultSet=True, modifiedRowCount=None, operationType=0, operationId=THandleIdentifier(secret='pq)"\xf9LNN\x00\x00\x00\x00\xb1Z\x17{', guid='pq)"\xf9LNN\x00\x00\x00\x00\xb1Z\x17{')), orientation=4, maxRows=100): TFetchResultsResp(status=TStatus(errorCode=None, errorMessage="File 'hdfs://devana1/user/hive/warehouse/precision_1/date=2019-02-01/000000_0_copy_1' column 'prec' has a scale that does not match the table metadata scale. File metadata scale: 2 Table metadata scale: 6\n", sqlState='HY000', infoMessages=None, statusCode=3), results=None, hasMoreRows=None)
kindly suggest
... View more
Labels:
- Labels:
-
Apache Hive
-
Apache Impala
07-27-2016
03:27 PM
@slachterman Understood, Thanks for the detailed explaination!!! What if i have to re import again and output directory already exist, is there options for sqoop import with overwrite option?
... View more
07-27-2016
02:34 PM
@slachterman Thanks it worked now!!! where can i find that data in HDFS if no path is specified ? So It has to be changed if i am importing from oracle/postgresql ?
... View more
07-27-2016
01:34 PM
Hi @slachterman when i am trying to run the sqoop import command i am getting the below error. Thanks, Ravikumar.
... View more
Labels:
- Labels:
-
Apache Sqoop
07-27-2016
07:13 AM
@Sindhu I tried using sqoop import command and it gives me the below error.
... View more
07-27-2016
06:51 AM
@Michael Young I tried even using lowercase p it couldn't connect me. But I just tried with ssh root@192.168.228.128 it works. Thank you
... View more
07-26-2016
03:09 PM
@Michael Young Do you mean i should use cygwin/putty to connect to sandbox and then try running these commands so that i can scroll thru my result log? I was trying to connect to sandbox using cygwin with below command and it does not recognize the command
... View more
07-26-2016
02:48 PM
Hi All, When I execute any command in sandbox, if my command has any syntax error/if something is going wrong while execution, it gives me whole list of error and it takes me to bottom of error how can i see the full error in sandbox from the begining. My intention is scroll thru the error and see each and every line of my error.
... View more