Monday, January 16, 2006

Splunk and syslog-ng

I setup splunk at the office today. Well, I should say I continued to set it up. Today's task was really about incorporating a centralized syslog-ng host with splunk. I'm here to tell you it was pretty easy. Here's what I did:

1. Download syslog-ng RPM here: RPM

2. Make some syslog-ng configuration changes like so:

[root@demetri05 root]# cat /etc/syslog-ng/syslog-ng.conf
# syslog-ng configuration file.
# This should behave pretty much like the original syslog on RedHat. But
# it could be configured a lot smarter.
# See syslog-ng(8) and syslog-ng.conf(5) for more information.
# 20000925
# Updated by Frank Crawford () - 10 Aug 2002
# - for Red Hat 7.3
# - totally do away with klogd
# - add message "kernel:" as is done with klogd.
# Updated by Frank Crawford () - 22 Aug 2002
# - use the log_prefix option as per Balazs Scheidler's email

options { sync (0);
time_reopen (10);
log_fifo_size (1000);
long_hostnames (off);
use_dns (no);
use_fqdn (no);
create_dirs (yes);
keep_hostname (yes);

source s_sys { pipe ("/proc/kmsg" log_prefix("kernel: ")); unix-stream ("/dev/log"); internal(); };
source r_src { tcp(ip("") port(514)); };

destination d_cons { file("/dev/console"); };
destination d_mesg { file("/var/log/messages"); };
destination d_auth { file("/var/log/secure"); };
destination d_mail { file("/var/log/maillog"); };
destination d_spol { file("/var/log/spooler"); };
destination d_boot { file("/var/log/boot.log"); };
destination d_cron { file("/var/log/cron"); };
destination d_mlal { usertty("*"); };
destination d_kernel { file("/var/log/kern"); };
destination d_remote { tcp("" port(514)); };

destination dr_cons { file("/dev/console"); };
destination dr_mesg { file("/var/log/remote-syslog-ng/$HOST/messages"); };
destination dr_auth { file("/var/log/remote-syslog-ng/$HOST/secure"); };
destination dr_mail { file("/var/log/remote-syslog-ng/$HOST/maillog"); };
destination dr_spol { file("/var/log/remote-syslog-ng/$HOST/spooler"); };
destination dr_boot { file("/var/log/remote-syslog-ng/$HOST/boot.log"); };
destination dr_cron { file("/var/log/remote-syslog-ng/$HOST/cron"); };
destination dr_mlal { usertty("*"); };
destination dr_kernel { file("/var/log/remote-syslog-ng/$HOST/kern"); };

filter f_filter1 { facility(kern); };
filter f_filter2 { level(info) and
not (facility(mail)
or facility(authpriv)
or facility(cron)
or program("kernel")); };
filter f_filter3 { facility(authpriv); };
filter f_filter4 { facility(mail); };
filter f_filter5 { level(emerg); };
filter f_filter6 { facility(uucp) or
(facility(news) and level(crit)); };
filter f_filter7 { facility(local7); };
filter f_filter8 { facility(cron); };
filter f_kernel { level(info) and program("kernel"); };

#log { source(s_sys); filter(f_filter1); destination(d_cons); };
log { source(s_sys); filter(f_filter2); destination(d_mesg); };
log { source(s_sys); filter(f_filter3); destination(d_auth); };
log { source(s_sys); filter(f_filter4); destination(d_mail); };
log { source(s_sys); filter(f_filter5); destination(d_mlal); };
log { source(s_sys); filter(f_filter6); destination(d_spol); };
log { source(s_sys); filter(f_filter7); destination(d_boot); };
log { source(s_sys); filter(f_filter8); destination(d_cron); };
log { source(s_sys); filter(f_kernel); destination(d_kernel); };

#log { source(r_src); filter(f_filter1); destination(dr_cons); };
log { source(r_src); filter(f_filter2); destination(dr_mesg); };
log { source(r_src); filter(f_filter3); destination(dr_auth); };
log { source(r_src); filter(f_filter4); destination(dr_mail); };
log { source(r_src); filter(f_filter5); destination(dr_mlal); };
log { source(r_src); filter(f_filter6); destination(dr_spol); };
log { source(r_src); filter(f_filter7); destination(dr_boot); };
log { source(r_src); filter(f_filter8); destination(dr_cron); };
log { source(r_src); filter(f_kernel); destination(dr_kernel); };
log { source(s_sys); destination (d_remote); };

# vim: syntax=syslog-ng

3. Edit the Splunk Indexing Live Files (Tailing Processor) like so:

I used the segment number method for specifying my hostname. There were other options as well.

Then I pointed my browser to my splunk box and started going at it.

Pretty neat.

Sphere: Related Content


jrichardson said...

How did you configure the tailing processor in splunk?

I'd like create a "rule" for "Everything in /var/log/remote/$HOSTNAME/messages" without having to create a seperate "file" tag for each remote server.

dmourati said...

Ok, so I've dropped the $HOST, or $HOSTNAME from my syslog-ng and splunk configs. Just makes it easier for me to add new hosts and get the logs in without any tinkering.

Anonymous said...

Is there an update to the doc for the splunk config that you reference? It doesn't seem to exist anymore, at least not at that address

dmourati said...

Sigh, docs come and go I guess.

Here's a newer, command line way of doing the same:

Alternatively, you can use the Admin Interface for Web-based Configuration:

Thanks for posting your question. By re-reading the docs I've just found a neat command line option I've needed for some time called --active-only. That saves me some trouble I'm having with old log files being resent.

Anonymous said...

we are offering best splunk online training with job support and high quality training facilities and well expert faculty . to Register you free demo please visit ,splunk training in hyderabad

nasreen basu said...

really cool post, highly informative and professionally written and I am glad to be a visitor of this perfect blog, thank you for this rare info!
splunk training in hyderabad
servicenow training in hyderabad