Community Articles
Find and share helpful community-sourced technical articles
Labels (1)
Super Guru

SysDig

SysDig (Github) is an open source tool that allows for the exploration, analysis and trouble shooting of Linux systems and containers. It is well documented and very easy to install and use. It can be used for container and Linux system diagnostics, security analysis, monitoring and basic system information capture. Remember that sysdig can produces thousands of lines of messages and can continue doing so forever depending on the options selected. Check out the examples and read through all the options, you can monitor a ton of data really fast and also check for security anomalies.

NiFi Ingesting SysDig

Sysdig can produce amazing amounts of logs. I chose to ingest 1 second chunks as ASCII JSON. I selected those options and listed them below. The results are arrays of JSON. I decided it's best to save them as a large JSON files for now and convert them to ORC later for Hive analysis in Zeppelin. You could also split them into individual JSON rows and process those. I also save them to Apache Phoenix for fast queries.

ExecuteProcess Command

sysdig -A -j -M 1 --unbuffered

I just wrap that in a shell script for neatness.

HDF 2.0 / NIFI 1.0.0 Flow

8853-sysdig2.png

Event JSON from SysDig

{"evt.cpu":6,"evt.dir":">","evt.info":"fd=7(<f>/usr/lib64/python2.7/lib-dynload/_elementtree.so) ","evt.num":111138,"evt.outputtime":1477313882635597873,"evt.type":"fstat",
"proc.name":"python","thread.tid":14602}

Apache Phoenix Table

CREATE TABLE sysdigevents
(
   evtcpu              varchar,
   evtdir            varchar,
   evtinfo                 varchar,
   evtoutputtime            varchar,
   evttype                varchar,
   procname                varchar,
   threadtid     varchar,
   evtnum             varchar not null primary key
);


Links


NiFi Flow

sysdig.xml

1,150 Views