Support Questions
Find answers, ask questions, and share your expertise

Hive SERDEPROPERTIES clarification

Highlighted

Hive SERDEPROPERTIES clarification

Contributor

could anybody tell what is the purpouse of code in create table statement

ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' WITH SERDEPROPERTIES ( 'input.regex' = '([^ ]*) - - \\[([^\\]]*)\\] "([^\ ]*) ([^\ ]*) ([^\ ]*)" (\\d*) (\\d*) "([^"]*)" "([^"]*)"', 'output.format.string' = "%1$$s %2$$s %3$$s %4$$s %5$$s %6$$s %7$$s %8$$s %9$$s")

a)The source log file is present in below link

http://hadooptutorial.info/wp-content/uploads/2014/11/hadooplogs.txt

could anybody tell what is the purpouse of code highlighted in bold letters in create table statement
 
CREATE EXTERNAL TABLE intermediate_access_logs ( ip STRING, date STRING, method STRING, url STRING, http_version STRING, code1 STRING, code2 STRING, dash STRING, user_agent STRING) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' WITH SERDEPROPERTIES ( 'input.regex' = '([^ ]*) - - \\[([^\\]]*)\\] "([^\ ]*) ([^\ ]*) ([^\ ]*)" (\\d*) (\\d*) "([^"]*)" "([^"]*)"', 'output.format.string' = "%1$$s %2$$s %3$$s %4$$s %5$$s %6$$s %7$$s %8$$s %9$$s") LOCATION '/user/hive/warehouse/original_access_logs';
3 REPLIES 3
Highlighted

Re: Hive SERDEPROPERTIES clarification

Guru

SERDE can tell hive how to get columns/fields out of a row of data. That statement tells hive ot use Regex SerDe and gives the regular expression that it needs to use to extract columns out.

Highlighted

Re: Hive SERDEPROPERTIES clarification

Contributor

Thanks for input.I need more information how we are extracting is not clear i mean in oracle

'[a-z] [0-9]' denotes lower-case letter followed by a space and a number.

I am looking for similar kind of information for

'input.regex'='([^ ]*) - - \\[([^\\]]*)\\] "([^\ ]*) ([^\ ]*) ([^\ ]*)" (\\d*) (\\d*) "([^"]*)" "([^"]*)"'

'output.format.string'="%1$$s %2$$s %3$$s %4$$s %5$$s %6$$s %7$$s %8$$s %9$$s"

Re: Hive SERDEPROPERTIES clarification

Contributor

Any inputs on my clarification will be appreciated?

Don't have an account?