Member since
03-06-2017
9
Posts
1
Kudos Received
3
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3541 | 04-14-2017 07:01 AM | |
3566 | 04-07-2017 08:02 AM | |
5129 | 03-06-2017 08:34 PM |
04-14-2017
07:01 AM
To set this in Impala to execute either as a SQL file or hue you would set the variables as shown in the first 2 lines below. Below is an example of using static partitioning and dynamic partitioning together where the EffDTM and FirstNameLetter are static partitioning and LastNameLetter uses dynamic partitioning. set var:hive.exec.dynamic.partition.mode=nonstrict; set var:hive.exec.dynamic.partition=true; INSERT OVERWRITE TABLE person_by_name PARTITION (EffDTM='201701', FirstNameLetter='a', LastNameLetter) firstname, lastname, .. .. lower(substr(lastname, 0, 1)) as LastNameLetter FROM person WHERE lower(substr(firstname, 0, 1)) = 'a';
... View more
04-01-2017
09:25 PM
So I was able to solve the issue but not using a Serde. What I did was load the fixedwidth file into a single string and then used substring to parse through the line as follows: CREATE TABLE FIXEDWIDTH_TEMP ( whole_line STRING ) LOCATION '/data/shared_workspace/test/fixedwidth_temp'; CREATE EXTERNAL TABLE FIXEDWIDTH_STG ( ZIP_CODE_3043 STRING, ZIP4_CODE_3044 STRING, PREFIX_TITLE_8093 STRING, STATE_3038 STRING, PHONE_3032 STRING, ... .. 1800 lines more ... ) LOCATION '/data/shared_workspace/test/fixedwidth_stg'; INSERT INTO FIXEDWIDTH_STG SELECT SUBSTRING(acxiom_line,1,5), SUBSTRING(acxiom_line,6,4), SUBSTRING(acxiom_line,10,1), SUBSTRING(acxiom_line,11,3), SUBSTRING(acxiom_line,14,2), SUBSTRING(acxiom_line,16,10), SUBSTRING(acxiom_line,26,12), . . .1800 lines more FROM FIXEDWIDTH_TEMP;
... View more
03-06-2017
08:34 PM
Argggg. Ok I need to find a wall and pound my head against it. The issue was I was running the first_etl.pig as pig -x local first_etl.pig which runs it locally expecting a local file and what I want is to run this on the Hadoop cluster. Running this as pig first_etl.pig fires this off and finds the file.
... View more