Splunk launches IT troubleshooting wiki | InfoWorld | News | 2006-04-03 | By China Martens, IDG News Service

I was quoted in a news article written by China Martins announcing the launch of Splunk Base.

One of the 3,500-plus users who has been beta testing Splunk Base is Demetri Mouratis. He's a systems administrator at Nuasis Corp., a vendor of Internet Protocol-based contact center software based in Mountain View, California.

Wading through log files contained on hundreds of servers to pin down the cause of a particular IT problem can take hours or an entire day, according to Mouratis. Typically, an administrator will have to log into each server manually to check out its log files. Using Splunk Base on a properly configured system "can greatly decrease the time," acting as a shortcut to help to resolve the issue, he said. "You don't have to cut and paste the query or go and Google (Profile, Products, Articles) it," he added.

For the full text of the article, please follow this link.

Splunk launches IT troubleshooting wiki | InfoWorld | News | 2006-04-03 | By China Martens, IDG News Service

Sphere: Related Content

04-01-06_1446.jpg

04-01-06_1446.jpg
Originally uploaded by dmourati.

balloon boy rwc

Sphere: Related Content

Splunk Integration With Xen

I've been dying to get my hands on a Fedora Core 5 for some time now. After a day of downloading, today I finally got to run my first FC 5 install. As expected, everything just works. I used CDs for my first install, but quickly setup the standard Red Hat/Fedora tree on my kickstart server. I'm glad I did.

The main draw for me with FC 5 is the integration with Xen. Up until now, I've had to use the live CD to try out Xen because I couldn't justify the time patching and compiling kernels to run inside Xen. With FC 5, all that work has disappeared. Here's a very good doc describing how to get started:

http://www.fedoraproject.org/wiki/FedoraXenQuickstartFC5

Now, I had one limitation on my development box, namely that it didn't have any outbound access to the 'net. That's okay, I setup squid on my workstation and was able to move on to configuring yum. I'm not sure what the problem was, but yum kept complaining about not being able to reach its repositiories. Whatever, I would think a default config file ought to have good default vaules in there but I guess not. After whacking that into shape I was able to install Xen on top of my fresh vanilla FC5. A quick edit to /etc/grub.conf (while I was talking to my Mom I might add, "Hi Mom) and I was all set to create domUs.

FC 5 provides a script to setup new domains automatically. This is nice.

Here's how I created one of my domUs, called xendomain2:

/usr/sbin/xenguest-install.py -n xendomain2 -f /home/xen/xendomain2 -r 256 -l http://kickstart.priv.nuasis.com/kickstart/fedora/core/5/i386/os/

This is okay, but I want an automated way. I don't like dealing with anaconda, enkay?

xenguest-install.py -n xendomain4 -f /home/xen/xendomain4 -s 25 -r 256 -l http://kickstart.priv.nuasis.com/kickstart/fedora/core/5/i386/os -x ks=http://kickstart.priv.nuasis.com/kickstart/cfgs/xen.cfg

Pow!

Here's whats going on at the moment on my xen test box:

[root@dev3-rep01 ~]# xm list
Name ID Mem(MiB) VCPUs State Time(s)
Domain-0 0 995 2 r----- 1103.9
xendomain2 13 256 1 ------ 39.2
xendomain1 14 256 1 -b---- 12.9
xendomain3 20 256 1 -b---- 13.7
xendomain4 25 256 1 r----- 481.4

[root@localhost ~]# rpm -qi kernel-xenU
Name : kernel-xenU Relocations: (not relocatable)
Version : 2.6.15 Vendor: Red Hat, Inc.
Release : 1.2054_FC5 Build Date: Tue 14 Mar 2006 02:22:51 PM PST
Install Date: Fri 31 Mar 2006 12:15:31 AM PST Build Host: hs20-bc1-3.build.redhat.com
Group : System Environment/Kernel Source RPM: kernel-2.6.15-1.2054_FC5.src.rpm
Size : 13268213 License: GPLv2
Signature : DSA/SHA1, Tue 14 Mar 2006 03:25:11 PM PST, Key ID b44269d04f2a6fd2
Packager : Red Hat, Inc.
Summary : The Linux kernel compiled for unprivileged Xen guest VMs
Description :
This package includes a version of the Linux kernel which
runs in Xen unprivileged guest VMs. This should be installed
both inside the unprivileged guest (for the modules) and in
the guest0 domain.


All above OSs are FC5 at the moment. I'm cool with that until I can figure out how to patch/install RHEL 3/RHEL 4 ontop of Xen on FC5.

I noticed some weirdness with ganglia, my open source cluster monitoring tool during and after the initial xendomain1 install. This is not surprising given all the virtual network and mac address forgery going on to support nesting OSs like this.

So, on to Splunk. I'm running v1.2.4 (thanks guys, for the quick turnaround on the installer). I wanted to capture these new logs coming out of the Xen install and configuration. I decidied to streamline my syslog-ng configuration on my splunk box a bit. Until now, each time I would add new machine, the syslogs wold end up under /var/log/remote-syslog-ng/$HOST/* where $HOST was the hostname/IP of the sending system. This is great for keeping logs seperate but sucks when it comes to modifying splunk configuration each time. So, I simplified my syslog-ng and have one "melting pot" for all my remote syslogs, regardless of originating system.

A few changes to my config and I was ready to restart syslog-ng.

That was fun!

Sphere: Related Content

Realer than real-deal Holyfield

This is the best article about the film I just saw Awesome, I Fucking Shot that.




I like wired, so I'm glad to see they covered this in the most detailed fashion. Interviewing MCA, aka Cap't Crunch, gives the story if full effect.

http://www.wired.com/wired/archive/14.04/play.html?pg=4

There were some other really cool effects that Yauch fails to mention. My favorites were the bass boom on Paul Revere and the total color wash out for a black and while nearly comic book effect.

And one more article, this one from the Sundance Review.

http://www.cinematical.com/2006/01/21/sundance-review-awesome-i-fuckin-shot-that/

Sphere: Related Content

Sourcefire Network Security - News & Events

I just got a very interesting email about the merger of Sourcefire and Checkpoint.

From: Jennifer Steffens
Update on Sourcefire Acquisition
2006-03-23 18:55

Hi Everyone,

We wanted to make everyone aware that Check Point and Sourcefire
withdrew their application today after carefully considering the
complexities of the CFIUS process, the lengthy ongoing delays in the
CFIUS process, and the current climate for international acquisition.

There is a press release with further details available at
http://www.sourcefire.com/news/press_releases/pr-13.html.

This in no way changes our commitment to the Snort technology or
community. If you have any questions, please let me know.

Thanks,
Jennifer



--
Jennifer S. Steffens
Director, Product Management - Snort
Sourcefire - Security for the Real World
W: 410.423.1930 | C: 202.409.7707
www.sourcefire.com | www.snort.org

Sourcefire Network Security - News & Events

Reading the above email and the corresponding press release,
it seems the reason the merger was called of was because of
complexities in the Committee on Foreign Investment in the
United States(CFIUS).

Here's another link on the aborted merger

http://www.informationweek.com/news/showArticle.jhtml;jsessionid=GCGILUQ1SVFLOQSNDBECKH0CJUMEKJVN?articleID=183702698

Sphere: Related Content

MercuryNews.com | 03/22/2006 | Dell buys Alienware, and AMD gets a foothold

MercuryNews.com | 03/22/2006 | Dell buys Alienware, and AMD gets a foothold

Sphere: Related Content

Network Monitoring Tools

Network Monitoring Tools

Sphere: Related Content

Hacking the Hack of Hacking Red Hat Kickstart

O'Reilly publishing has released a series of books called the Hacks Series. They define hack as "A clever solution to an interesting problem."

I'm a big fan.

I've read:

Google Hacks
Linux Server Hacks
Linux Server Hacks, Volume Two (Electric Boogaloo)
Mind Performance Hacks (just bought this one yesterday in fact)
Network Security Hacks

They're all really cool and eminently useful books.

Last Friday, I was assigned the task of creating a single CD based install of Red Hat Enterprise Linux for the purpose of unattended installation (Kickstart). This is opposed to the option of physically swapping all four CDs as released by Red Hat. Until now, we, and our customers, had been leveraging Kickstart using the stock RHEL Update 6 CD 1 to do network based installations. In my opinion, going over the network is more elegant than hacking up a custom CD any day. However, our "Partner" got our "Marketing Department" to agree that we would provide CD media for the entire installation of our product. As such, the network approache, while technically greatly superior, would have to take a back seat.

My starting point was an excellent article written by Brett Schwarz for Linux Journal. Here's the article link:

http://www.linuxjournal.com/article/6473

This article, unfortunately, is based on Red Hat 8.0, which is now obsoleted and past its end of life. Luckily, other Red Hat Enterprise customers have updated the doc with their tweaks to support all the latest RHEL distros. In this post, I will consolidate all the work done previously by Brett and add in the fixes/tweaks as added by the community. This follows the O'Reilly approach covered in their Hacks series "Hacking the Hack."

As Slick Rick says, "Here we Go."

First, get your hands on the 4 CDs for the RHEL release in question. For me, this was RHEL ES3 Update 6. Here are the CDs and their md5sums. (Yes, I know RHEL 3 Update 7 is out, so is RHEL 4 Update 3, and Fedora Core 5.) Again, full credit to Brett for all of these steps. They've just been tweaked as necessary to support newer RH releases and documented end-to-end here.

[root@kickstart iso]# ls -lah rhel-3-u6-i386-es-disc*
-rw-r--r-- 1 root root 152M Sep 21 11:54 rhel-3-u6-i386-es-disc1.iso
-rw-r--r-- 1 root root 627M Sep 21 11:43 rhel-3-u6-i386-es-disc2.iso
-rw-r--r-- 1 root root 637M Sep 21 11:48 rhel-3-u6-i386-es-disc3.iso
-rw-r--r-- 1 root root 282M Dec 6 16:38 rhel-3-u6-i386-es-disc4.iso

[root@kickstart iso]# md5sum rhel-3-u6-i386-es-disc*
2a695a0dc773b2172b35f8164b10f2f3 rhel-3-u6-i386-es-disc1.iso
68e7b2f34cb1903c24da02e25bcf5462 rhel-3-u6-i386-es-disc2.iso
8aa48608434065fb481d462d8495583c rhel-3-u6-i386-es-disc3.iso
3b35b450ecec27c5a9c63300f7518d3f rhel-3-u6-i386-es-disc4.iso

[root@kickstart iso]# mkdir Update6
[root@kickstart iso]# mkdir -p CD{1,2,3,4}
[root@kickstart iso]# mkdir onecd

Let's mount these badboys, and I don't want to hear any shit about the FHS right now either.

mount -o loop /kickstart/iso/rhel-3-u6-i386-es-disc1.iso /kickstart/ES3/Update6/CD1/
mount -o loop /kickstart/iso/rhel-3-u6-i386-es-disc2.iso /kickstart/ES3/Update6/CD2/
mount -o loop /kickstart/iso/rhel-3-u6-i386-es-disc3.iso /kickstart/ES3/Update6/CD3/
mount -o loop /kickstart/iso/rhel-3-u6-i386-es-disc4.iso /kickstart/ES3/Update6/CD4/

Copy the RPMS:

cp -a CD1/* onecd/
cp -a CD{2,3,4}/RedHat/RPMS/* onecd/RedHat/RPMS/

Now the tricky part, coming up with a complete set of RPMs that fits on one CD and is internally consistent. Let's use Brett's python scripts to get this started.

cd /kickstart/ES3/Update6/onecd/

getGroupPkgs.py comps.xml > /kickstart/ES3/Update6/pkglist
syncRpms.py onecd/RedHat/RPMS/ pkglist > pkgs_rem

(Note, I had to make a few changes to the syncRPMS.py script to reflect the newer arcitecture) Here's the modified script:

#!/usr/bin/python

#
# Removes packages that are not part of a package list from
# a given directory. This is used to remove RPMs that are
# not needed in a custom distro. The package list specified
# is just a text file with a list of package names, one per
# line.
#
# syncRpms.py /some/path/to/rpms/ /tmp/pkglist
#
# Copyright (C) 2003 Brett Schwarz (brett_schwarz@yahoo.com)
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA
# 02111-1307 USA
#
# Version History
# ---------------
# 0.2 08-04-2003 Added fix from Alain Tauch for accepting
# /path instead of /path/
# 0.1 22-03-2003 Original release

import rpm
import sys, os, glob

if len(sys.argv) != 3:
sys.stderr.write("Usage\n")
sys.stderr.write("%s \n" % (sys.argv[0],))
sys.exit(1)

tgtdir = sys.argv[1]
pkglist = sys.argv[2]

#
# get pkg name to file name mapping
#
name2file = {}
ts = rpm.TransactionSet("", rpm._RPMVSF_NOSIGNATURES)
for fname in glob.glob(tgtdir + '/*.rpm'):
fd = os.open(fname, os.O_RDONLY)
hdr = ts.hdrFromFdno(fd)
name2file[hdr[rpm.RPMTAG_NAME]] = fname
os.close(fd)

#
# Read in packages from package list
#
fd = open(pkglist, "r")
pkgs = {}
for l in fd.readlines():
if l[-1]=='\n':
l = l[:-1]
pkgs[l] = 1

fd.close()

#
# Remove unwanted packages
#
for n, f in name2file.items():
if not pkgs.has_key(n):
os.remove(f)
print n

#
# Check to see if there are pkgs not in tgt dir
#
for p in pkgs.keys():
if not name2file.has_key(p):
print "Package not in tgt dir: ", p

OK. Now to test the resultant set. Again, cool trick Brett.

mkdir /tmp/testdb
rpm --initdb --dbpath /tmp/testdb
rpm --test --dbpath /tmp/testdb -Uvh *.rpm

For whatever reason, Brett's script didn't do a very good job of gettin the right set together. No matter, the rpm command tells you about failed dependencies so all you have to do is go locate the missing RPMs and copy them into the oncecd tree. I ended up building up a third file called pkgsadd and then ran:

for i in $(cat pkgsadd); do src=$(find CD* -name $i); /bin/cp -f $src onecd/RedHat/RPMS/; done

Eventually, the set was consistent.
[root@kickstart RPMS]# rpm --test --dbpath /tmp/testdb -Uvh *.rpm
warning: acl-2.2.3-1.i386.rpm: V3 DSA signature: NOKEY, key ID db42a60e
warning: package glibc = 2.3.2-95.37 was already added, replacing with glibc <= 2.3.2-95.37 warning: package kernel = 2.4.21-37.EL was already added, replacing with kernel <= 2.4.21-37.EL warning: package kernel-smp = 2.4.21-37.EL was already added, replacing with kernel-smp <= 2.4.21-37.EL Preparing... ########################################### [100%]

One more test, for good measure: rpm -K *.rpm | grep "NOT *OK" Now, here's where my steps differed from the initial doc.

Pay attention.

First, you need anaconda and anaconda-devel.

up2date anaconda anaconda-runtime
export PYTHONPATH=/usr/lib/anaconda
/usr/lib/anaconda-runtime/pkgorder /kickstart/ES3/Update6/onecd/ i386 > /kickstart/ES3/Update6/onecd/RedHat/base/pkgorder
/usr/lib/anaconda-runtime/genhdlist --withnumbers --fileorder /kickstart/ES3/Update6/onecd/RedHat/base/pkgorder --hdlist /kickstart/ES3/Update6/onecd/RedHat/base/hdlist /kickstart/ES3/Update6/onecd/

Now, I'm going to add all the ks.cfg files to the CD repository.

[root@kickstart onecd]# http://satellite.priv.nuasis.com/kickstart/ks/label/ncc-3.0
[root@kickstart onecd]# wget http://satellite.priv.nuasis.com/kickstart/ks/label/ncc-3.0-ide
[root@kickstart onecd]# wget http://satellite.priv.nuasis.com/kickstart/ks/label/ncc-3.0-md

This almost bit me:

[root@kickstart CD1]# cp .discinfo /kickstart/ES3/Update6/onecd/

One more fix I had to do. My ks.cfgs out there on the satellite serve all point to a network based install. To make this new kickstart go off of the cdrom, I need to point to cdrom instead of url inside each of ncc-3.0, ncc-3.0-ide, and ncc-3.0-md.

The big step:

[root@kickstart root]# mkisofs -r -T -J -V "ncc-3.0-rhel-3-u6-i386-es.t1" -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table -v -o /kickstart/iso/ncc-3.0-rhel-3-u6-i386-es.t1.iso /kickstart/ES3/Update6/onecd

154528 extents written (301 MB)

301 MB, not bad.
[root@kickstart root]# /usr/lib/anaconda-runtime/implantisomd5 /kickstart/iso/ncc-3.0-rhel-3-u6-i386-es.t1.iso
Inserting md5sum into iso image...
md5 = 12bf4c82d5d59b854e00d343b02d7cc6
Setting supported flag to 0

Cool.

Sphere: Related Content