This flow utilizes the PartitionRecord processor and
Record Reader/Writer controller services (introduced in NiFI 1.2)
to parse the NiFi app log in Grok format, convert to JSON format and
group the output by log level (INFO, WARN, ERROR).
855d9cd4-015d-1000-8dc5-71f1f96a4779
PartitionRecord_GroktoJSON
a5f1cf52-d390-3976-0000-000000000000
451a635e-0803-3845-0000-000000000000
1 GB
10000
451a635e-0803-3845-0000-000000000000
f755d278-0d5d-3752-0000-000000000000
PROCESSOR
0 sec
1
info
0
b9769ff7-9474-3d16-0000-000000000000
451a635e-0803-3845-0000-000000000000
1 GB
10000
451a635e-0803-3845-0000-000000000000
004f06d1-2225-3864-0000-000000000000
PROCESSOR
0 sec
1
warn
0
d4318ed1-667f-309f-0000-000000000000
451a635e-0803-3845-0000-000000000000
1 GB
10000
454.9999999999999
451.9633397129244
454.9999999999999
501.9633397129244
451a635e-0803-3845-0000-000000000000
6c2b5fbd-9993-3466-0000-000000000000
PROCESSOR
0 sec
1
failure
0
f51f6830-4283-35fa-0000-000000000000
451a635e-0803-3845-0000-000000000000
1 GB
10000
451a635e-0803-3845-0000-000000000000
28d7416c-58f2-3dbc-0000-000000000000
PROCESSOR
0 sec
1
error
0
ff1e4b87-4228-3a21-0000-000000000000
451a635e-0803-3845-0000-000000000000
1 GB
10000
451a635e-0803-3845-0000-000000000000
8e439369-8abd-33e5-0000-000000000000
PROCESSOR
0 sec
1
success
0
50e6a8cf-b093-3cbb-0000-000000000000
451a635e-0803-3845-0000-000000000000
1 GB
10000
451a635e-0803-3845-0000-000000000000
ee3bbc30-8af1-3b74-0000-000000000000
PROCESSOR
0 sec
1
success
0
521e4b69-b2f6-3cae-0000-000000000000
451a635e-0803-3845-0000-000000000000
1 GB
10000
451a635e-0803-3845-0000-000000000000
6c2b5fbd-9993-3466-0000-000000000000
PROCESSOR
0 sec
1
success
0
e1d48f9f-ef95-3ac5-0000-000000000000
451a635e-0803-3845-0000-000000000000
nifi-record-serialization-services-nar
org.apache.nifi
1.3.0
Schema Write Strategy
Schema Write Strategy
schema-access-strategy
schema-access-strategy
schema-registry
org.apache.nifi.schemaregistry.services.SchemaRegistry
schema-registry
schema-name
schema-name
schema-text
schema-text
Date Format
Date Format
Time Format
Time Format
Timestamp Format
Timestamp Format
Pretty Print JSON
Pretty Print JSON
JsonRecordSetWriter
false
Schema Write Strategy
schema-name
schema-access-strategy
schema-name
schema-registry
01fe7c27-dec5-320d-0000-000000000000
schema-name
${schema.name}
schema-text
${avro.schema}
Date Format
Time Format
Timestamp Format
Pretty Print JSON
true
ENABLED
org.apache.nifi.json.JsonRecordSetWriter
01fe7c27-dec5-320d-0000-000000000000
451a635e-0803-3845-0000-000000000000
nifi-registry-nar
org.apache.nifi
1.3.0
nifi-logs
nifi-logs
AvroSchemaRegistry
false
nifi-logs
{
"type": "record",
"name": "nifi_logs",
"fields": [
{ "name": "timestamp", "type": "string" },
{ "name": "level", "type": "string" },
{ "name": "thread", "type": "string" },
{ "name": "class", "type": "string" },
{ "name": "message", "type": "string" },
{ "name": "stackTrace", "type": "string" }
]
}
ENABLED
org.apache.nifi.schemaregistry.services.AvroSchemaRegistry
1bb1d190-8746-3f5b-0000-000000000000
451a635e-0803-3845-0000-000000000000
nifi-record-serialization-services-nar
org.apache.nifi
1.3.0
schema-access-strategy
schema-access-strategy
schema-registry
org.apache.nifi.schemaregistry.services.SchemaRegistry
schema-registry
schema-name
schema-name
schema-text
schema-text
Grok Pattern File
Grok Pattern File
Grok Expression
Grok Expression
no-match-behavior
no-match-behavior
GrokReader
false
schema-access-strategy
schema-name
schema-registry
01fe7c27-dec5-320d-0000-000000000000
schema-name
${schema.name}
schema-text
${avro.schema}
Grok Pattern File
Grok Expression
%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} \[%{DATA:thread}\] %{DATA:class} %{GREEDYDATA:message}
no-match-behavior
append-to-previous-message
ENABLED
org.apache.nifi.grok.GrokReader
cf8cbced-9c51-3722-0000-000000000000
451a635e-0803-3845-0000-000000000000
583.9605495852537
43.45155132629043
222ba91c-e908-3e81-0000-000000000000
cf8cbced-9c51-3722-0000-000000000000
1 GB
10000
cf8cbced-9c51-3722-0000-000000000000
9dde9644-32e1-34fa-0000-000000000000
PROCESSOR
0 sec
1
success
0
4ef3684c-7625-3ca0-0000-000000000000
cf8cbced-9c51-3722-0000-000000000000
1 GB
10000
cf8cbced-9c51-3722-0000-000000000000
34c921bf-849b-3ed8-0000-000000000000
PROCESSOR
0 sec
1
success
0
9dde9644-32e1-34fa-0000-000000000000
cf8cbced-9c51-3722-0000-000000000000
11141.252679403602
1605.644713328848
nifi-standard-nar
org.apache.nifi
1.3.0
WARN
1
Log Level
Log Level
Log Payload
Log Payload
Attributes to Log
Attributes to Log
Attributes to Ignore
Attributes to Ignore
Log prefix
Log prefix
ALL
false
30 sec
Log Level
warn
Log Payload
false
Attributes to Log
Attributes to Ignore
Log prefix
0
0 sec
TIMER_DRIVEN
1 sec
LogAttribute - WARN
true
success
org.apache.nifi.processors.standard.LogAttribute
ee6fbd7f-9c75-3da5-0000-000000000000
cf8cbced-9c51-3722-0000-000000000000
11342.248371137215
1326.8055428430816
nifi-standard-nar
org.apache.nifi
1.3.0
WARN
1
File Size
File Size
Batch Size
Batch Size
Data Format
Data Format
Unique FlowFiles
Unique FlowFiles
generate-ff-custom-text
generate-ff-custom-text
ALL
false
30 sec
File Size
0B
Batch Size
1
Data Format
Text
Unique FlowFiles
true
generate-ff-custom-text
0
5 sec
TIMER_DRIVEN
1 sec
GenerateFlowFile
false
success
org.apache.nifi.processors.standard.GenerateFlowFile
34c921bf-849b-3ed8-0000-000000000000
cf8cbced-9c51-3722-0000-000000000000
11569.68820867292
1605.644713328848
nifi-standard-nar
org.apache.nifi
1.3.0
WARN
1
Log Level
Log Level
Log Payload
Log Payload
Attributes to Log
Attributes to Log
Attributes to Ignore
Attributes to Ignore
Log prefix
Log prefix
ALL
false
30 sec
Log Level
error
Log Payload
false
Attributes to Log
Attributes to Ignore
Log prefix
0
0 sec
TIMER_DRIVEN
1 sec
LogAttribute - ERROR
true
success
org.apache.nifi.processors.standard.LogAttribute
Generate Warnings & Errors
8e439369-8abd-33e5-0000-000000000000
451a635e-0803-3845-0000-000000000000
0.0
209.6226774177221
nifi-update-attribute-nar
org.apache.nifi
1.3.0
WARN
1
Delete Attributes Expression
Delete Attributes Expression
Store State
Store State
Stateful Variables Initial Value
Stateful Variables Initial Value
schema.name
schema.name
ALL
false
30 sec
Delete Attributes Expression
Store State
Do not store state
Stateful Variables Initial Value
schema.name
nifi-logs
0
0 sec
TIMER_DRIVEN
1 sec
Set schema.name = nifi-logs
false
success
org.apache.nifi.processors.attributes.UpdateAttribute
bd0efd4c-3086-3c99-0000-000000000000
451a635e-0803-3845-0000-000000000000
0.0
0.0
nifi-standard-nar
org.apache.nifi
1.3.0
WARN
1
tail-mode
tail-mode
File to Tail
File to Tail
Rolling Filename Pattern
Rolling Filename Pattern
tail-base-directory
tail-base-directory
Initial Start Position
Initial Start Position
File Location
File Location
tailfile-recursive-lookup
tailfile-recursive-lookup
tailfile-rolling-strategy
tailfile-rolling-strategy
tailfile-lookup-frequency
tailfile-lookup-frequency
tailfile-maximum-age
tailfile-maximum-age
ALL
false
30 sec
tail-mode
Single file
File to Tail
./logs/nifi-app.log
Rolling Filename Pattern
tail-base-directory
Initial Start Position
Beginning of File
File Location
Local
tailfile-recursive-lookup
false
tailfile-rolling-strategy
Fixed name
tailfile-lookup-frequency
10 minutes
tailfile-maximum-age
24 hours
0
5 sec
TIMER_DRIVEN
1 sec
TailFile
false
success
org.apache.nifi.processors.standard.TailFile
ee3bbc30-8af1-3b74-0000-000000000000
451a635e-0803-3845-0000-000000000000
0.0
619.1684273969818
nifi-standard-nar
org.apache.nifi
1.3.0
WARN
1
Routing Strategy
Routing Strategy
error
error
info
info
warn
warn
ALL
false
30 sec
Routing Strategy
Route to Property name
error
${log_level:equals('ERROR')}
info
${log_level:equals('INFO')}
warn
${log_level:equals('WARN')}
0
0 sec
TIMER_DRIVEN
1 sec
RouteOnAttribute
false
error
false
info
true
unmatched
false
warn
org.apache.nifi.processors.standard.RouteOnAttribute
f755d278-0d5d-3752-0000-000000000000
451a635e-0803-3845-0000-000000000000
598.9605495852537
452.12036418516345
nifi-standard-nar
org.apache.nifi
1.3.0
WARN
1
Log Level
Log Level
Log Payload
Log Payload
Attributes to Log
Attributes to Log
Attributes to Ignore
Attributes to Ignore
Log prefix
Log prefix
ALL
false
30 sec
Log Level
info
Log Payload
true
Attributes to Log
Attributes to Ignore
Log prefix
0
0 sec
TIMER_DRIVEN
1 sec
LogAttribute
true
success
org.apache.nifi.processors.standard.LogAttribute
004f06d1-2225-3864-0000-000000000000
451a635e-0803-3845-0000-000000000000
598.9605495852537
619.1684273969818
nifi-standard-nar
org.apache.nifi
1.3.0
WARN
1
Log Level
Log Level
Log Payload
Log Payload
Attributes to Log
Attributes to Log
Attributes to Ignore
Attributes to Ignore
Log prefix
Log prefix
ALL
false
30 sec
Log Level
info
Log Payload
true
Attributes to Log
Attributes to Ignore
Log prefix
0
0 sec
TIMER_DRIVEN
1 sec
LogAttribute
true
success
org.apache.nifi.processors.standard.LogAttribute
28d7416c-58f2-3dbc-0000-000000000000
451a635e-0803-3845-0000-000000000000
598.9605495852537
777.4409208257885
nifi-standard-nar
org.apache.nifi
1.3.0
WARN
1
Log Level
Log Level
Log Payload
Log Payload
Attributes to Log
Attributes to Log
Attributes to Ignore
Attributes to Ignore
Log prefix
Log prefix
ALL
false
30 sec
Log Level
info
Log Payload
true
Attributes to Log
Attributes to Ignore
Log prefix
0
0 sec
TIMER_DRIVEN
1 sec
LogAttribute
true
success
org.apache.nifi.processors.standard.LogAttribute
6c2b5fbd-9993-3466-0000-000000000000
451a635e-0803-3845-0000-000000000000
0.0
411.9633397129244
nifi-standard-nar
org.apache.nifi
1.3.0
WARN
1
record-reader
org.apache.nifi.serialization.RecordReaderFactory
record-reader
record-writer
org.apache.nifi.serialization.RecordSetWriterFactory
record-writer
log_level
log_level
ALL
false
30 sec
record-reader
1bb1d190-8746-3f5b-0000-000000000000
record-writer
e1d48f9f-ef95-3ac5-0000-000000000000
log_level
/level
0
0 sec
TIMER_DRIVEN
1 sec
PartitionRecord
false
failure
true
original
false
success
org.apache.nifi.processors.standard.PartitionRecord
08/28/2017 15:18:42 EDT