Hi, I have a stream of data coming in to hdfs. I want to store the data in to hive. --------------------------------------------------------------------------------------- Sample data:(data is in single line but with multiple attributes) sample=data1 _source="/s/o/u" destination="/d/e/s" _ip="0.0.0.0" timestamp=20170802 10:00:00 text="sometext_with$spec_char" sample=data2 destination="/d/e/s" _ip="0.0.0.0" timestamp=20170802 10:00:00 text="sometext_with$spec_char" _source="/s/o/u" technology="r"o"b"ust" sample=data3 _ip="0.0.0.0" timestamp=20170802 10:00:00destination="/d/e/s" text="sometext_with$spec_char" _source="/s/o/u" --------------------------------------------------------------------------------------- Problems with data: 1.data do not follow same order if you can see (sample_data 1 has source, destination, timestamp, text. sample_data2 has destination,timestamp,text, source e.t.c) 2. the attributes dont follow same convention (_source, destination, _ip, timestamp,text etc; but basically with "_" and with out "_". 3. the attributes are not fixed (sample_data1 has source, destination,timestamp,text; sample_data2 has destination, _ip, timestamp,text,source and technology) sample | source| destination | ip | text | technology | data1 |a/b/c | /d/e/s | 0.0.0.0 |sometext_with$spec_char | NULL| data2 |a/b/c | /d/e/s | 0.0.0.0 |sometext_with$spec_char | r"o"b"ust data3 |a/b/c | /d/e/s | 0.0.0.0 |sometext_with$spec_char | NULL| Thanks for your support ... View more

Hi , I have stream data (GetHDFS will be running continuosly ) which contains number of lines. e.g: <start>this is 123_@":text coming from [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] linedelimited. A stream of above lines of data will be in file I have to extract text from above message [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] I tried using a extract text processor and used custom property extract: ([hdfs.*log"]). I tried the above in java regex evaluator, it shows correct text extracted. but when I run the flow, output gets the complete text. expected: [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] actual : <start>this is 123_@":text coming from [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] linedelimited. Please help me to correct the regex to extract correct text. ... View more