Problem with assigning values to attributes in NiFi

[#1]

Hello everyone,

I have a problem with assigning values from Squid logs to attributes in FlowFile.
My process consists of:
→ ListenTCP (listening to Squid logs),
→ ReplaceText (deleting a symbol from a given field),
→ UpdateAttribute (assigning the schema to the log data),
→ ConvertRecord (data separation and adding columns),
→ UpdateAttribute (assigning values from columns to attributes in FlowFile).

The components listed above are only part of the entire process. At the end of the process, the processed data from the logs should be fed to the Hive database using PutHiveQL.

The configurations of the listed components are as follows.

With this configuration, Content is correct, but in the Attributes tab, the attribute name is "Empty string set" and "No value set". System attributes such as filename, path or record.count have valid values.

[#2]

More screenshots did not fit in the post, so here I add the Attribues and Content tabs

[#3]

@Wojtek

I believe you are misunderstanding how the UpdateAttribute processor functions.

-

Each new property you add expects a property name (this becomes name of attribute being created or updated) and a value. The value can be a string or a NiFi Expression Language (EL) statement.

-

In your screenshot above you have created EL statements. For example

Property = bytes
Value = ${bytes}

What the above will actually do is:

- The EL statement "${bytes}" tells NiFi to try to locate an NiFi attribute (At no time does the updateAttribute processor read the content of the FlowFile) with a property name of bytes and return its assigned value. That returned value will then be used to change the existing value assigned to the FlowFile attribute "bytes" or create a new FlowFile attribute with property name "bytes" and assign the value to that.

-

NiFi searches for NiFi Attributes in the following hierarchy:

1. NiFi checks all existing attributes assigned already to FlowFile being processed.

2. NiFi checks all in scope process group variables

3. NiFi checks the NIFi variable registry file

4. NiFi checks the NiFi JVM properties

5. NiFi checks the NIFi user system environment variables.

-

Since the attribute "bytes" does not exsit in any of these places you are ending up with no value or empty string values being set on all these new properties.

-

Since you are trying to extract values from the content and assign those to FlowFile attributes, you will want to use a different processor. Perhaps ExtractText instead.

-

Thank you,

Matt

-

If you found this answer addressed your question, please take a moment to login in and click the "ACCEPT" link.