Skip to main content
mark_hadoop
mark_hadoop
Gold ⭐️
August 1, 2017
Solved

How to Extract Text from FlowFiles in Apache NiFi Using ExtractText

Hi ,

I have stream data (GetHDFS will be running continuosly ) which contains number of lines.

e.g:

<start>this is 123_@":text coming from [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] linedelimited.

A stream of above lines of data will be in file

I have to extract text from above message

[hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"]

I tried using a extract text processor and used custom property

extract: ([hdfs.*log"]).

I tried the above in java regex evaluator, it shows correct text extracted. but when I run the flow, output gets the complete text.

expected: [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"]

actual : <start>this is 123_@":text coming from [hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"] linedelimited.

Please help me to correct the regex to extract correct text.

MattWho
MattWhoAnswer
Titanium ⭐️⭐️⭐️
August 1, 2017

@Hadoop User

Your Java regular expression needs to escape the "[" and "]" since they have reserved meaning in Java.

Try using the following java regular expression instead:

(\[hdfs.*log"\])

Thanks,

Matt

View in thread

7 replies

MattWho
Titanium ⭐️⭐️⭐️
MattWhoAnswer
Titanium ⭐️⭐️⭐️
August 1, 2017

@Hadoop User

Your Java regular expression needs to escape the "[" and "]" since they have reserved meaning in Java.

Try using the following java regular expression instead:

(\[hdfs.*log"\])

Thanks,

Matt

Translate
mark_hadoop
mark_hadoopAuthor
Gold ⭐️
August 1, 2017

@Matt Clarke I have used your suggestion, but result is same, it fetches the complete line instead of [hdfs....... .log"]

for clarification I will let you know the steps which I am following

1. GetHDFS

2. Splittext: count-1.

3. Extract text:

  1. (\[hdfs.*log"\])

4. Update Attribute

5. PutHDFS

not sure why it is pulling complete line?

Thanks

Translate
MattWho
Titanium ⭐️⭐️⭐️
MattWho
Titanium ⭐️⭐️⭐️
August 1, 2017

@Hadoop User

The ExtractText processor will extract the text that matches your regex and assign it to an attribute matching the property name on the FlowFile. The content of the FlowFile remains unchanged. Then you update a FlowFiles Attribute and finally use PutHDFS to write the content (which at this time you have not changed at all) to HDFS.

If your intent is to write the modified string to HDFS, you need to update the actual content of the FlowFile and nit just create and modify attributes. For that use case, you would want to use ReplaceText processor instead.

You would configure ReplaceText similar to the following:

23384-screen-shot-2017-08-01-at-122929-pm.png

The above will result in the actual content of the FlowFile being changed to:

[hdfs file="/a/b/c" and' the; '''', "", file is streamed. The location=["/location"] and log is some.log"]

Thanks,

Matt

Translate
    Terms & ConditionsAccessibility statement