Sunday, March 19, 2006

Splunk Integration With OpenSSH

This blog post covers Splunk Integration with OpenSSH. OpenSSH is the lifeblood of any distributed Linux system. I practically live inside and SSH session. During any given day, I may SSH into 20 different system all around the world. As such, it is important to keep track of what SSH is doing. What better way than with Splunk?

This integration actually picks up after some work I did earlier integrating Splunk with Syslog-ng. For more information on how I'm setup, please read my earlier post

Splunk and syslog-ng

Now that I have remote syslog enabled, I can do all kinds of cool stuff. This is especially useful for troubleshooting large networks of interconnected systems where SSH is undoubtedly already in play. My syslog-ng configuration is already being processed by Splunks tailingprocesser which is looking at all the stock syslog-ng files plus per host syslog-ng directories for remote syslog. Here' s a look at my central syslog-ng configuration:

[root@demetri05 log]# find remote-syslog-ng/

To get a sense of just how pervasive SSH is in my environment, take a look at the following splunk. This shows that in the past day, there are over 10,000 events relating to SSH. Wow. So far I've uploaded all the standard SSH messages I could generate. That turned out to be around 20 types or so. Now, I'm going to make some changes to my SSH server to increase its logging level. Take a look at the /etc/ssh/sshd_config file for reference:

# Logging
#obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
SyslogFacility AUTHPRIV
#LogLevel INFO

You'll notice a large uptick in the number of events starting in the 12:00 PM hour today. That's becaue I've changed the LOGLEVEL configuration in my /etc/ssh/sshd_config file to DEBUG. WARNING, don't do this on any system with real users. According to the manpage "Logging with a DEBUG level violates the privacy of users and is not recommended." This is my system and I'm the only user so I can proceed but I won't leave this in place beyond my work today for reasons of privacy and also to keep my log files manageable.

Here's a look at the type of events generated by the inceased LOGLEVL in SSHD. This time I'm splunking for sshd debug1.
That's kind of cool, actually. Now I have a whole set of new event types to upload to SplunkBase. Again, the fact remains that real systems shouldn't be configured with this level of logging for SSH but I'll make sure to note that in all my event descriptions once they're uploaded. I'm going to simplify my work by sorting the events into event types and then uploading each event type on up to SplunkBase for integration. That way I'll be sure to get all the event types up there without worrying about duplicating log submissions. At the moment, I see 36 event types related to SSH and the new DEBUG loglevel I've put in place.
I'm drilling down on each of these event types, then uploading the corresponding log entry up to splunk base one at a time. It seems like this is uncharted territory as far as SplunkBase is concerned. Each of these event types are new which is cool.

As I go, I'm keeping an eye on the total number of event types I've submitted to SplunkBase by watching my splunk user page here:

At the moment, I'm at 383 total event types submitted and counting. Completing this SSH debug logging should push the total above 400 types.

Now I'm going to reset the LOGLEVL on my SSH configuration to turn off all this DEBUG level logging.
That in turn generated a bunch more event types moving the total from 36 to 43. Hmm. Let me add those for good measure. The new event types are easy enough to spot because they each have only one associated event.

Here's a link to the last event type from my SSH DEBUG testing:

Sphere: Related Content


Anonymous said...

we are offering best splunk online training with job support and high quality training facilities and well expert faculty . to Register you free demo please visit ,splunk training in hyderabad