Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Ingest timestamped apache logs with NIFI

Highlighted

Ingest timestamped apache logs with NIFI

Explorer

Since some weeks we manage to replace all our logstash config by nifi dataflow.

I got problems ton ingest some Apache Logs

The logs are formatted like that : error.log.$TIMESTAMP and access.log.$TIMESTAMP. The logs can rotate more than one time a day changing name with a new timestamp.

I try to use “tailfile” but it doesn’t work (or I don’t know how to configure it for working well). Does anyone got an idea to resolve it ?

2 REPLIES 2
Highlighted

Re: Ingest timestamped apache logs with NIFI

David, do you mean that there's no 'error.log' or 'access.log' files at all, but only the ones with the timestamp suffix? Can you give more details on the issue you are running into?

Basically, you should be able to use the TailFile processor, set the Start Offset to the Beginning of Time (so it captures rolled files too) and configure the filename pattern of the processor to be something like 'access.log.*'.

Highlighted

Re: Ingest timestamped apache logs with NIFI

You can also take a look at some pre-cooked templates here https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates

Don't have an account?
Coming from Hortonworks? Activate your account here