And You Don't Stop, a dmourati Production: Splunk Integration With OpenSSH

This blog post covers Splunk Integration with OpenSSH. OpenSSH is the lifeblood of any distributed Linux system. I practically live inside and SSH session. During any given day, I may SSH into 20 different system all around the world. As such, it is important to keep track of what SSH is doing. What better way than with Splunk?

This integration actually picks up after some work I did earlier integrating Splunk with Syslog-ng. For more information on how I'm setup, please read my earlier post

Splunk and syslog-ng

Now that I have remote syslog enabled, I can do all kinds of cool stuff. This is especially useful for troubleshooting large networks of interconnected systems where SSH is undoubtedly already in play. My syslog-ng configuration is already being processed by Splunks tailingprocesser which is looking at all the stock syslog-ng files plus per host syslog-ng directories for remote syslog. Here' s a look at my central syslog-ng configuration:

[root@demetri05 log]# find remote-syslog-ng/
remote-syslog-ng/
remote-syslog-ng/demetri04
remote-syslog-ng/demetri04/secure
remote-syslog-ng/demetri04/messages
remote-syslog-ng/demetri04/cron
remote-syslog-ng/demetri04/boot.log
remote-syslog-ng/demetri04/maillog
remote-syslog-ng/demetri04/kern
remote-syslog-ng/demetri05
remote-syslog-ng/demetri05/boot.log
remote-syslog-ng/demetri05/cron
remote-syslog-ng/demetri05/secure
remote-syslog-ng/demetri05/messages
remote-syslog-ng/demetri05/maillog
remote-syslog-ng/demetri05/kern
remote-syslog-ng/splunk
remote-syslog-ng/splunk/demetri04-messages
remote-syslog-ng/splunk/demetri05-messges

To get a sense of just how pervasive SSH is in my environment, take a look at the following splunk. This shows that in the past day, there are over 10,000 events relating to SSH. Wow. So far I've uploaded all the standard SSH messages I could generate. That turned out to be around 20 types or so. Now, I'm going to make some changes to my SSH server to increase its logging level. Take a look at the /etc/ssh/sshd_config file for reference:

# Logging
#obsoletes QuietMode and FascistLogging
#SyslogFacility AUTH
SyslogFacility AUTHPRIV
#LogLevel INFO

You'll notice a large uptick in the number of events starting in the 12:00 PM hour today. That's becaue I've changed the LOGLEVEL configuration in my /etc/ssh/sshd_config file to DEBUG. WARNING, don't do this on any system with real users. According to the manpage "Logging with a DEBUG level violates the privacy of users and is not recommended." This is my system and I'm the only user so I can proceed but I won't leave this in place beyond my work today for reasons of privacy and also to keep my log files manageable.

Here's a look at the type of events generated by the inceased LOGLEVL in SSHD. This time I'm splunking for sshd debug1.
That's kind of cool, actually. Now I have a whole set of new event types to upload to SplunkBase. Again, the fact remains that real systems shouldn't be configured with this level of logging for SSH but I'll make sure to note that in all my event descriptions once they're uploaded. I'm going to simplify my work by sorting the events into event types and then uploading each event type on up to SplunkBase for integration. That way I'll be sure to get all the event types up there without worrying about duplicating log submissions. At the moment, I see 36 event types related to SSH and the new DEBUG loglevel I've put in place.
I'm drilling down on each of these event types, then uploading the corresponding log entry up to splunk base one at a time. It seems like this is uncharted territory as far as SplunkBase is concerned. Each of these event types are new which is cool.

As I go, I'm keeping an eye on the total number of event types I've submitted to SplunkBase by watching my splunk user page here:

http://www.splunk.com/user/dmourati

At the moment, I'm at 383 total event types submitted and counting. Completing this SSH debug logging should push the total above 400 types.

Now I'm going to reset the LOGLEVL on my SSH configuration to turn off all this DEBUG level logging.
That in turn generated a bunch more event types moving the total from 36 to 43. Hmm. Let me add those for good measure. The new event types are easy enough to spot because they each have only one associated event.

Here's a link to the last event type from my SSH DEBUG testing:

http://www.splunk.com/event/SP-CAAAC5H