how to perform Log file analysis in hadoop ?

[#1]

log1.txt

Hi,

I have some sever logs. I want to perform this log analysis using PIG and HIVE. I am attaching sample log file.

It would be great if someone help me to discover different patterns from the logs.

Thank You.

[#2]

@heta desai Have a look at this link. You can use the Hive table structure(s) given according to your log file format and process them as needed. The key is using Regex for parsing the records into individual columns. The tutorial talks about using HBase, but you can skip it if you don't want to use it at this point of time.

Let know if you need any help.

[#3]

@heta desai

Did the answer help in the resolution of your query? Please close the thread by marking the answer as Accepted!