Splunk Light Forwarder

In this instalment of Fun With Splunk, I will walk you through how to setup a distributed Splunk installation using the new Splunk Light Forwarder. While lightweight forwarding has been around for some time, the new install makes things a bit more streamlined. The idea here is that you setup on machine to index your data and run the web UI. This box should be free from other production duties. An infrastructure box, such as one running Nagios, Ganglia, Cacti or the like is a good candidate.

The logs you will want to analyse, however, are likely on your production machines, your web servers, app servers, databases, etc. Since those machines have real work to do, you'd like to offload the data indexing and searching to a machine not in the critical path of your customers. The new Light Forwarder is what you want.

First, download Splunk for your architecture. I'm running version 3.4.6, the latest as of March 4, 2009, on x86_64 for Linux 2.6 kernels. I downloaded one copy to my workstation to get started.

Next, I scp the RPM file to all three of my machines for this install, two Light Forwarders, and one full-blown Indexer.

Run "rpm -Uvh /home/dmourati/splunk-3.4.6-51113-linux-2.6-x86_64.rpm" on all three machines to install the binaries. Next, setup splunk to auto start out of init. To do this, I run "/opt/splunk/bin/splunk enable boot-start" again on all three machines. Finally, start splunk with the newly installed init script via "/etc/init.d/splunk start".

When I attempted to connect to the indexer with my web browser, I noticed some firewall ports issues. I opened port 8000 for the UI, 9997 for the intra-splunk data transfer and was able to connect. On the forwarders, you can open the port but you won't need it in the long run. Remember to close the port back down in your firewall config or at least document why you have 8000 open. There will be no listener on the Light Forwarders when you are done.

Next, I consulted the new 3.4.6 Documentation on setting up Light Forwarders. I've already configured my Indexer to receive data. So I'm on to the next step.

Then, on the forwarding Splunk instance:

1. Install Splunk on the machine that will be forwarding data.

2. Enable data forwarding by pointing your forwarder at the receiver using these instructions.

"/opt/splunk/bin/splunk add forward-server indexer:9997"

3. Use Splunk Web or the CLI to add inputs as described here. Data from these inputs will be sent via the forwarder to the receiver.

I'm using follow-only, which I helped drive as a Splunk feature in a previous life.

# /opt/splunk/bin/splunk add monitor /var/log/eyefi -follow-only True
Added monitor of '/var/log/eyefi'.
# /opt/splunk/bin/splunk add monitor /var/log/httpd -follow-only True
Added monitor of '/var/log/httpd'.
# /opt/splunk/bin/splunk add monitor /var/log/messages -follow-only True
Added monitor of '/var/log/messages'.

4. Then, use Splunk Web or the CLI to enable Splunk forwarder or light forwarder.

"# /opt/splunk/bin/splunk enable app SplunkLightForwarder"

Now, restart splunk on the Light Forwarders

Point your broswer to the Indexer at port 8000 and start Splunking.

Sphere: Related Content