Need assistance about huge number data transfer to Syslog through Apache NiFi

Muhyid — Sat, 27 Jun 2020 14:13:45 GMT

Dear All,

I need your suggestion huge number data transfer to Syslog through Apache NiFi

My aim is transferring all IIS logs (located on Azure Blob Storage) to syslog line by line. Therefore I am using split text for parsing lines. Extract Text is transferring split line on the attribute, in this way I can say to syslog processor "Message Body: IISHttp${msg}". I will explain details at the below

Actually my problem is bottleneck on the Extract Text. I have to transfer IIS Logs near-real time (less than 1 hour delay). But Extract Text isn't draining the messages (flow files) from the queue in proper time. I tried increasing Thread Number, Run Duration, increasing/reducing Queue size but I couldn't achieve my target. The queue between split text and extract text are always full. I have log gap about 13 hours.

I am trying find a way for that

My flow is that;

1. Getting IIS Log files from Azure Blob Storage

1.1. List Azure Blob Storage Processor

1.2. Route on Attribute Processor (I have date filter RegEx on it)

1.3. Fetch Azure Blob Storage Processor

2. Split Text Processor: Splitting each IIS Log File to line by line with Split Text Processor.

2.1. Line Split Count: 1

2.2. Maximum Fragment Size: No value set

2.3. Header Line Count: 0

2.4. Header Line Marker Characters: No value set

2.5. Remove Trailing Newlines: True

3. Extract Text Processor: Transferring new flow files which is produced by Split Text Processor to Extract Text Processor. Extract Text Processor is the problematic point

3.1. All Properties are Default

3.2. I added one RegEx in the Properties. I would like to carry on Flow Files attributes to Syslog

3.2.1. Property Name: msg

3.2.2. Value: (.*)

4. Put Syslog Processor: Transferring all flow files where is coming from Extract Text to Put Syslog Processor.

4.1. All Properties are Default or configured properly for requirements (such as IP address of the Syslog, port etc.)

4.2. Message Body: IISHttp${msg}

4.3. There is no flow file waiting on the Put Syslog Processor queue (between extract text and put syslog).

I tried those options also;

1. Route Text Processor usage instead of Extract Text. But I failed

1.1. Removed Extract Text Processor

1.2. Added Route Text but I didn't accomplish to transfer line by line to syslog.

2. Put TCP Processor usage instead of Put Syslog. But I couldn't complete the configuration my knowledge is not enough for that.

2.1. Removed Split Text Processor

2.2. Removed Extract Text Processor

2.3. Removed Syslog Processor

2.4. Added Put TCP Processor

2.4.1. Hostname: Syslog Server

2.4.2. Port: Syslog Server Port (TCP)

2.4.3. Outgoing Message Delimiter: \n (for splitting each line from entire IIS Log file. I want to have just 1 line to syslog transfer for each time)

2.4.4. SSL Context Service --> StandardRestrictedSSLContextService (configuring for mutual authentication)

2.4.5. Rest of the Properties are default

2.5. I need to add some prefixes to each line which is produced by \n delimiter for Syslog Server. How will I do these?

2.5.1. Each Line should be begin these prefixes:

2.5.1.1. Message Timestamp: ${now():format('MMM d HH:mm:ss')} --- Default Property of Put Syslog

2.5.1.2. Message Hostname: ${hostname(true)} --- Default Property of Put Syslog

2.5.2. After these two prefix I need to append IISHttp (Message Body: IISHttp ${msg}) wording.

2.5.3. I don't want to rewrite or completely replace each line. I just want to append some things

2.5.3.1. For Example:

My IIS Log File line like this:

2020-03-13 13:59:19 XXX-YYY GET /Maintenance/Status.svc X-ARR-LOG-ID=267ed22c-f1b 200 0 0 1005 1086 46

My line will be like this:

Jun 26 23:29:09 SERVER1 IISHttp 2020-03-13 13:59:19 XXX-YYY GET /Maintenance/Status.svc X-ARR-LOG-ID=267ed22c-f1b 200 0 0 1005 1086 46

I guess, Put TCP Processor is better way but I can't continue to configuration due to lack of my knowledge

Do you have any comment or suggestion?

My environment details are below:

Apache NiFi 1.11.3 (Single Node, not cluster)

Windows Server 2016

Java JRE 1.8.0_241 (64 Bit)

Re: Need assistance about huge number data transfer to Syslog through Apache NiFi

Muhyid — Sat, 11 Jul 2020 17:55:47 GMT

Dear All,

I solved problem myself.
I used Update Text and Put Tcp processors for that. It is working properly and giving better performance

question Need assistance about huge number data transfer to Syslog through Apache NiFi in Support Questions

Need assistance about huge number data transfer to Syslog through Apache NiFi

Re: Need assistance about huge number data transfer to Syslog through Apache NiFi