question Re: regex pattern for hive regex serde in Support Questions

regex pattern for hive regex serde

raja_aravapalli — Tue, 02 Aug 2016 00:35:34 GMT

Can someone pls help me creating regex pattern to use for creating a hive table with RegEx Serde....

I want hive regex table to be created with the pattern ^^^^^^^^^^ [10 anchor characters]  as a delimiter! I am not sure what would be the regex pattern of hive table for this!!

please help.

Thanks a lot in advance

Re: regex pattern for hive regex serde

mqureshi — Tue, 02 Aug 2016 00:43:33 GMT

@Raja A

I will try \\^ for each character. so basically

\\^\\^\\^\\^\\^\\^\\^\\^\\^\\^

Can you try this?

Re: regex pattern for hive regex serde

raja_aravapalli — Tue, 02 Aug 2016 01:52:32 GMT

Thanks for the response @mqureshi

I Tried below two ways, but that did not work...

"input.regex" = "\\^\^\\^\\^\\^\\^\\^\\^\\^\\^"
"input.regex" = "(\\^\^\\^\\^\\^\\^\\^\\^\\^\\^)"

Any other thoughts i can try of ?

Re: regex pattern for hive regex serde

mqureshi — Tue, 02 Aug 2016 02:12:44 GMT

Hi @Raja A

In both cases the second one has only single slash. Can you try it with two slashes. I think the first one should work if you just add two slashes. Also, why not try to make it work with only "^^" first. Figure out how to make ^^ work and then you can simply extend that to ^^^^^^^^^^.

Re: regex pattern for hive regex serde

raja_aravapalli — Tue, 02 Aug 2016 04:05:27 GMT

Thanks for your response and time @mqureshi

I tried for ^^... its still not working...!! And, sorry, my above comment had a typo !!

Below is what i am trying !!

create external table <table_name> ( col1 string, col2 string )
ROW FORMAT SERDE "org.apache.hadoop.hive.contrib.serde2.RegexSerDe"
WITH SERDEPROPERTIES (
"input.regex" = "^^"
)
STORED AS TEXTFILE
LOCATION "<hdfs_path>";

Re: regex pattern for hive regex serde

mqureshi — Tue, 02 Aug 2016 04:10:11 GMT

@Raja A

I will try this on my machine when I get a chance, hopefully later today but I think it should

"input.regex"="^\\^\\^"

The first one signifies the beginning of a string and other two are for your matching pattern.

Re: regex pattern for hive regex serde

pminovic — Fri, 05 Aug 2016 08:36:14 GMT

Hi @Raja A, can you try

"input.regex"="(.*)\\^\\^\\^\\^\\^\\^\\^\\^\\^\\^(.*)"

You need groups to map to your table fields.

Re: regex pattern for hive regex serde

raja_aravapalli — Thu, 29 Sep 2016 16:50:56 GMT

Hi @Predrag Minovic

Thanks a lot for the response. this works!!

However, this is working only when i am having 2 columns in the input... but not working when there are more columns !!

Re: regex pattern for hive regex serde

pminovic — Thu, 29 Sep 2016 17:21:46 GMT

Well, you said you had only 2 columns 🙂 For more columns you can either change the regex, or try MultiDelimitSerDe if you are on Hive-0.14 or newer. By the way, inspired by your question I wrote an article about RegexSerDe.