how to send logs from nifi to graylog

[#1]

Hello Team,

I am trying to send logs to the gaylog server from Nifi for the transfer of each file log.

could you please help me with the stepwise procedure to do so?

Also i have been trying to copy few files from remote server 1 to 2 and then the same files from 1 will be moved to 3 with the logs of each files in getting pushed to graylog.

can you please help me with the step wise procedure to perform this action.

i have been struggling from weeks to do above two task. 

[#2]

@NafeesShaikh93, Welcome to our community! To help you get the best possible answer, I have tagged in our Nifi experts @cotopaul @SAMSAL @BennyZ @MattWho @steven-matison who may be able to assist you further.

Please feel free to provide any additional information or details about your query, and we hope that you will find a satisfactory solution to your question.

[#3]

Thank you so much vidyasagar for responding back.

As a latro member, we are planning to use nifi for file transmission between one server to another remote server.
i have done the testing of file transmission using getsftp and putsftp processors. Now we are willing to get the logs of these files those have travelled from source to destination server. We want to store those logs into graylog application so that we can observe it over there. and we have tried Alot to push the logs to there, but so far we are not successful in this task.
could you please help us with the proper stepwise procedure to complete this job 

[#4]

hi @NafeesShaikh93,

What logs are you trying to push into Graylog? The logs generated by NiFi when transferring the files from Source to Target, or are you talking about other logs?

What have you tried and why did it not succeed? Let's start with that, as your Flow Logic was built specific for your use case and somebody else's logic might not suite your needs.

[#5]

Hello Cotopaul,

i am trying to push the logs of file information, like:

1- filename:

2- file type

3- size

4- source

5- destination

6- size at source

7- size at destination.

general category logs.

would it be fine with you to hop in a call to have more clarification on this topic.

i may be able to explain you in a better way.

[#6]

Now, based on your Flow Logic, where do you encounter problems and what are those problems?

In addition, how do you retrieve dynamically the destination and the size at destination? As far as I know, PutFile does not write those attributes so you will have to manually define them in your Flow and I do not see any UpdateAttribute or UpdateContent on your canvas. The size, the file type, the filename and the source are written by default when the file is extracted and your can further use them with the help of the NiFi Expression Language.

[#7]

@NafeesShaikh93 

Interesting Use case you have.  I am not all that familiar with all the methods the Graylog offers for ingesting logs from other servers.  I'd assume Syslog is one of them?  If so, NiFi offers. putSyslog processor.

Looking at the dataflow you build thus far, I am not sure what you are trying to accomplish.  The LogAttribute and logMessage processors allows you to write a log entry in a NiFi log defined by an appender and logger in the logback.xml NiFi configuration file.  By default these log lines would end up in the nifi-app.log.    You could however add an additional appender and the a custom logger to send log lines produced by these processors classes to the new appender thus isolating them from the other logging in the nifi-app.log.  There is no way to setup a specific logger by processor on canvas.  So every logAttribute and logMessage processor you use will write to the same destination NiFi appender log.

The classes for the logAttribute and logMessage processors are:

org.apache.nifi.processors.standard.LogAttribute
org.apache.nifi.processors.standard.LogMessage

NiFi also has a tailFile processor that can tail a log file and create FlowFiles with that log entries at content.  You could then use PutSyslog processor to send those log lines to yoru Graylog server possibly.

The above design involves extra disk I/O that may not be necessary since you could possibly design your flow to create FlowFile attributes with all the file information you want to send to GrayLog an then use a replaceText at end of successful dataflow to replace the content of yoru FlowFile with a crafted syslog formatted content from those attributes and send directly to Graylog via the PutSyslog processor.  This removes the need to write to a new logger and consume from that new log before sending o syslog.  But again this is a matter of preference.  Perhaps in you case maybe you want a local copy of these logs as well.

If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped.

Thank you,

Matt

[#8]

• well this is certainly new for me. I think it will be better if we could connect for 30 mints in our free time to have a clear cut understanding of what needs to be done to fix it.